USING THE XBOX KINECT TO DETECT FEATURES OF THE FLOOR SURFACE By STEPHANIE COCKRELL Submitted in partial fulfillment of the requirements For the degree of Master of Science Thesis Advisor: Gregory Lee Department of Electrical Engineering and Computer Science CASE WESTERN RESERVE UNIVERSITY May, 2013 CASE WESTERN RESERVE UNIVERSITY SCHOOL OF GRADUATE STUDIES We hereby approve this thesis/dissertation of _________Stephanie Cockrell_________ candidate for the Master of Science degree *. (signed) _______Gregory Lee_________ ___________Frank Merat___________ ________M. Cenk Cavusoglu________ (date) ___December 12, 2012__________ * We also certify that written approval has been obtained for any proprietary material contained therein. Table of Contents List of Tables ................................................................................................................................... v List of Figures ................................................................................................................................. vi Acknowledgements ......................................................................................................................... ix Abstract ............................................................................................................................................ x Chapter 1: Introduction .................................................................................................................... 1 Chapter 2: Literature Review ........................................................................................................... 5 Smart wheelchairs and obstacle detection ................................................................................... 5 Kinect ........................................................................................................................................... 6 Drivable surfaces ......................................................................................................................... 7 Uneven surfaces and ramps ......................................................................................................... 8 Chapter 3: Characterization of the Kinect ...................................................................................... 11 Software and hardware .............................................................................................................. 12 Distance limitations ................................................................................................................... 12 Distortion ................................................................................................................................... 14 Mounting angle .......................................................................................................................... 18 Selection of angle and height ..................................................................................................... 24 Chapter 4: Calculating the transform ............................................................................................. 27 Best-fit plane .............................................................................................................................. 27 Compute a rotation matrix ......................................................................................................... 27 Error ........................................................................................................................................... 28 Repeatability .............................................................................................................................. 31 Chapter 5: Classifying features of the floor surface ...................................................................... 35 Creating a height map ................................................................................................................ 36 Identifying bumps and obstacles ................................................................................................ 42 Identifying level floors and ramps ............................................................................................. 46 Results ........................................................................................................................................ 52 Speed .......................................................................................................................................... 63 Summary .................................................................................................................................... 65 Chapter 6: Conclusions and Future Work ...................................................................................... 67 Appendix A: Comparison of different floors ................................................................................. 69 Appendix B: Selection of points for calibration ............................................................................ 74 iii Estimate the slope at each data point ......................................................................................... 75 Obtain an estimate for the angle of the Kinect ........................................................................... 79 Calculate the Kinect’s height ..................................................................................................... 80 Accuracy .................................................................................................................................... 81 Bibliography .................................................................................................................................. 83 iv List of Tables Table 1: Technical specifications for the Kinect. .......................................................................... 11 Table 2: Rms errors for the region used for calibration and for the entire field of view. .............. 30 Table 3: Rms errors for 16 data sets with repect to their best-fit planes. ....................................... 32 Table 4: Time required to run each of the 6 parts of the algorithm. .............................................. 64 Table 5: Statistics on the data returned for different types of floors. ............................................. 70 Table 6: Comparing different values for the parameter “smoothConstant”, to find the result closest to the best-fit plane. ............................................................................................................ 78 Table 7: Evaluating the rms error for planes calculated by three different methods. .................... 82 v List of Figures Figure 1: Sample output from the Kinect’s depth camera and RGB camera. ................................ 11 Figure 2: Depth image of a hand 20 cm away from the Kinect. .................................................... 13 Figure 3: Depth image of a hand 60 cm away from the Kinect. .................................................... 13 Figure 4: Depth image and RGB image in a long hallway, showing the Kinect has a limitation in seeing long distances. ....................................................................................................... 14 Figure 5: Distortion due to the camera is very limited.. ................................................................ 15 Figure 6: Same data as the previous figure, but the figure has been rotated. Distortion due to the camera is very limited. ...................................................................................................... 16 Figure 7: Same data as the previous figure, but the figure has been rotated. Distortion due to the camera is very limited. ...................................................................................................... 17 Figure 8: The Kinect’s field of view for the data in the previous three images. ............................ 17 Figure 9: Depth image and RGB image of a room with a white floor. The Kinect’s line of sight is parallel to the floor surface. ........................................................................................... 18 Figure 10: Red and white floor. The Kinect is mounted at a 20 degree angle from the horizontal. .......................................................................................................................................... 19 Figure 11: Red and white floor. The Kinect is mounted at a 30 degree angle from the horizontal. .......................................................................................................................................... 19 Figure 12: Red and white floor. The Kinect is mounted at a 40 degree angle from the horizontal. .......................................................................................................................................... 19 Figure 13: Red and white floor. The Kinect is mounted at a 50 degree angle from the horizontal. .......................................................................................................................................... 20 Figure 14: Red and white floor. The Kinect is mounted at a 60 degree angle from the horizontal. .......................................................................................................................................... 20 Figure 15:Black floor. The Kinect is mounted at a 20 degree angle from the horizontal. ............. 21 Figure 16: Black floor. The Kinect is mounted at a 30 degree angle from the horizontal. ...........