Computer Aided Drafting, Design and Manufacturing Volume 24, Number 1, March 2014, Page 1 CADDM 3D Reconstruction by Kinect Sensor：A Brief Review LI Shi-rui1,2, TAO Ke-lu1,2, WANG Si-yuan1,2, LI Hai-yang1,2,CAO Wei-guo1, LI Hua 1 1. Key Laboratory of Intelligent Information Processing, Institute of Computing Technology Chinese Academy of Sciences, Beijing 100190, China; 2. University of Chinese Academy of Sciences, Beijing 100190, China. Abstract: While Kinect was originally designed as a motion sensing input device of the gaming console Microsoft Xbox 360 for gaming purposes, it’s easy-to-use, low-cost, reliability, speed of the depth measurement and relatively high quality of depth measurement make it can be used for 3D reconstruction. It could make 3D scanning technology more accessible to everyday users and turn 3D reconstruction models into much widely used asset for many applications. In this paper, we focus on Kinect 3D reconstruction. Key words: Kinect; 3D reconstruction; RGBD 1 Introduction map at 30Hz. Its sensor consists of an infrared laser projector, an infrared camera and an RGB camera. When Microsoft launched Kinect in North America Kinect as a scanner has many advantages: on November 4, 2010, Kinect was sold through an average of 133333 units per day, for a total of 8 million units in its first 60 days on sale, which has been confirmed by Guinness World Records as the fast-selling consumer electronics device ever[1]. During the last three years after the launch of the Microsoft Kinect, it became a hot research topic in computer vision. Kinect becomes a good available device in simultaneous localization and mapping (SLAM), 3D scene reconstruction, object recognition, augment reality, object tracking and motion analysis etc. Fig.1. The impact of the Microsoft Kinect in the field of 3D reconstruction is significant: Over the last three years, While Kinect was originally designed as a motion over 2000 papers related to the Microsoft Kinect 3D sensing input device of the gaming console Microsoft reconstruction have been published in renowned journals Xbox 360 for gaming purposes, it’s low-cost, and proceedings. reliability, speed of the depth measurement and relatively high quality of depth measurement make it (1) Simultaneously generate depth and color at to be able to be used for 3D reconstruction. An standard video rate; easy-to-use and low-cost scanning solution could make 3D scanning technology more accessible to (2) Using infrared, the depth sensor does not everyday users and turn 3D shape models into much interfere with visual spectrum to some extent; widely used asset for many applications, for instance (3) Low-cost, reliability and speed of measurement; in online shopping, gaming, movie, augment reality and so on. The impact of the Microsoft Kinect in the (4) Dense reconstruction. field of 3D reconstruction is shown in Fig.1. On the other hand, Kinect is not perfect as a Kinect shown in Fig.2 is an application of scanner because of fellow drawbacks: structured light. It generates an 11 bit 640*480 depth Corresponding author: LI Shi-rui, Male, Ph.D. candidate, E-mail: [email protected]. 2 Computer Aided Drafting, Design and Manufacturing (CADDM) , Vol.24, No.1, Mar. 2014 (1) Low x/y resolution and depth accuracy; fuse the RGB and depth information. The purpose of Kinect calibration is to get more accurate 3D (2) Depth map contains of numerous “holes” where measurement and to correctly align the RGB camera is no measurement; output and depth output. (3) Depth measurement is noisy especially in the Kinect calibration includes calibration of color edge of object; camera intrinsics, depth camera intrinsics and relative (4) Limitation of view; pose between the cameras. As far as we known almost all Kinect calibration methods[5-10] use almost the (5) Completely fail on transparent and specular [4] surface which are very common to everyday same principal of Zhang calibration method . The household object. calibration object is based on checkerboard. [4] Because of those advantages and disadvantages of The basic steps of Zhang are the following. First it, Kinect is being well studied in computer vision extract the checkerboard corner in intensity image. when it was launched. Until now, we found some And the 3D coordinates of checkerboard corner in other survey papers[2,3] to introduce Kinect-related Cartesian coordinate system of checkerboard is known. research. The major difference between our paper and Then utilize the camera geometrical model to project [2,3] is that we focus on Kinect 3D reconstruction 3D coordinate of checkerboard corner into intensity topic and try to give more detailed about this topic, image. The calibration method aims to minimize the while other papers intends to introduce all weighted sum of squares of the reprojection error Kinect-related topic, such as motion analysis, object measurement. recognition and so on. Zhang’s calibration method[4] was originally used The rest of the paper is organized as follows: Firstly, for ordinary color camera calibration and Kinect depth we introduce Kinect calibration in section 2. Then we camera calibration has its own characteristics. Firstly, we cannot find checkerboard corner in depth image. discuss the accuracy analysis of Kinect sensor in [5] section 3. In section 4, we present some methods do Smisek use infrared image instead of depth image to depth map preprocessing. In section 5, we will review calibrate the relative pose between color and depth many 3D reconstruction algorithms based-on Kinect camera. Infrared image with speckles projected by senor. We will give some experiment results in section infrared laser projector cannot directly used to extract 6. At last, we summarize the remained challenges of checkerboard corner. They generate Infrared image by 3D reconstruction based on Kinect sensor, and major a halogen lamp with the infrared laser projector trends in this exciting domain. blocked. They considered the shift between infrared image and depth Image. Meanwhile Herrera[6,7] and its extension work Raposo[9] utilize the planarity of calibration object. They let user manually selects the four corners of the calibration plane to obtain an initial guess. Otherwise, the depth internal model is different to ordinary camera. The raw data provided by Kinect depth camera is a disparity image. The calibration methods[5,6,7,9] give a same internal model to translate the disparity image to depth image. With the above calibration procedure the residuals of plane fitting showing a fixed-pattern noise on depth images from different distances. To compensate for this residual Fig.2. Kinect hardware includes an infrared projector, error, Smisek[5] formed a z-correction image of z infrared camera and an RGB camera. The raw data values constructed as the pixel-wise mean of all generated by Kinect includes color image, depth image and [6,7] infrared image. residual images and Herrera considered a more complicated spatial distortion pattern of depth. [7] 2 Kinect Calibration Herrera adds an external high quality color camera to improve the quality of depth map and In fact, Kinect has been calibrated during accuracy of pose estimation calibration. Yamazoe[8] manufacturing and the calibrated parameters are proposes a depth measurement error model stored in the device’s memory. Kinect use those considering both intrinsic parameters of the camera default parameters to get 3D points measurement and and the projector. LI Shi-rui et al., 3D Reconstruction by Kinect Sensor：A Brief Review 3 3 Accuracy Analysis of Kinect Depth Data map. In this way, a much smoother result can be obtained. And another paper[15] also proposes an The accuracy decides the application area of the adaptive occlusion-filling algorithm for depth map Kinect. The accuracy of Kinect depth data consists of processing and for restoration of scene backgrounds two part depth resolution and random error. using depth map. To improve depth maps, Qi[16] The principle of Kinect measurement is proposes a novel fusion based inpainting method. [17] substantially a stereo measurement. The raw data Moreover, Schmeing uses edge information found generated by Kinect is disparity measurement. A in the corresponding color stream via superpixel disparity measurement is stored as 11-bit integers in segmentation and compute a new representative depth Kinect. So the resolution of depth measurement which map which stores robust edge information. The depth is the distance between the two consecutive recorded map is then used to enhance the source depth map. values decreases with increasing distance to the sensor. And using region growing and Bilateral Filter [18] Smisek[5] gives a function of the depth z to calculate enhances depth image for Kinect . This method can depth resolution. At the distance of 0.5m, the depth significantly improve the quality of depth maps. [22] resolution is 0.65mm which is the optimal accuracy of Another similar method joins denoising and Kinect, and at the distance of 5m the depth resolution interpolation of depth maps for Microsoft Kinect is 71.37mm. sensors. Without using ancillary data like a color image, multiple aligned depth maps and something [11] Khoshelham theoretically analyses the factors else, we can exploit patchwise scene self-similarity influencing the accuracy of the data based on the across depth such as repetition of geometric primitives mathematical model of depth measurement. They or object symmetry by identifying and merging patch considered that inadequate calibration and inaccurate correspondences within the input depth map itself[19]. measurement of disparities are two main reasons for Hu[20] enhances depth by filtering using the Sobel depth error. Inadequate calibration can be solved by operator and the Laplacian operator. Another adequate calibration. Inaccurate measurement of complement[21] of the depth estimate within the Kinect disparities caused by correlation algorithm and is making use of a cross-modal stereo path obtained round-off error is most likely of a random error.