Real-Time Embedded Panoramic Imaging for Spherical Camera System
Total Page:16
File Type:pdf, Size:1020Kb
Real-time Embedded Panoramic Imaging for Spherical Camera System Main Uddin-Al-Hasan This thesis is presented as part of Degree of Bachelor of Science in Electrical Engineering Blekinge Institute of Technology September 2013 Blekinge Institute of Technology School of Engineering Department of Electrical Engineering Supervisor: Dr. Siamak Khatibi Examiner: Dr. Sven Johansson 2 3 Abstract Panoramas or stitched images are used in topographical mapping, panoramic 3D reconstruction, deep space exploration image processing, medical image processing, multimedia broadcasting, system automation, photography and other numerous fields. Generating real-time panoramic images in small embedded computer is of particular importance being lighter, smaller and mobile imaging system. Moreover, this type of lightweight panoramic imaging system is used for different types of industrial or home inspection. A real-time handheld panorama imaging system is developed using embedded real- time Linux as software module and Gumstix Overo and PandaBoard ES as hardware module. The proposed algorithm takes 62.6602 milliseconds to generate a panorama frame from three images using a homography matrix. Hence, the proposed algorithm is capable of generating panorama video with 15.95909365 frames per second. However, the algorithm is capable to be much speedier with more optimal homography matrix. During the development, Ångström Linux and Ubuntu Linux are used as the operating system with Gumstix Overo and PandaBoard ES respectively. The real-time kernel patch is used to configure the non-real- time Linux distribution for real-time operation. The serial communication software tools C- Kermit, Minicom are used for terminal emulation between development computer and small embedded computer. The software framework of the system consist UVC driver, V4L/V4L2 API, OpenCV API, FFMPEG API, GStreamer, x264, Cmake, Make software packages. The software framework of the system also consist stitching algorithm that has been adopted from available stitching methods with necessary modification. Our proposed stitching process automatically finds out motion model of the Spherical camera system and saves the matrix in a look file. The extracted homography matrix is then read from look file and used to generate real-time panorama image. The developed system generates real-time 180° view panorama image from a spherical camera system. Beside, a test environment is also developed to experiment calibration and real-time stitching with different image parameters. It is able to take images with different resolutions as input and produce high quality real-time panorama image. The QT framework is used to develop a multifunctional standalone software that has functions for displaying real-time process algorithm performance in real-time through data visualization, camera system calibration and other stitching options. The software runs both in Linux and Windows. Moreover, the system has been also realized as a prototype to develop a chimney inspection system for a local company. Keywords: Panorama Image, Image stitching, Image registration, SURF, Real-time computing, Gumstix Overo COM, PandaBoard ES, Embedded Linux, RTLinux, Real-time kernel, Embedded development, OpenCV, FFMPEG, V4L2, Computer Vision, QT framework 4 5 Acknowledgement I would like to thank and express love to all scientist and philosophers from ancient age, middle age and modern age, whose works given me ground to work on this area of science and technology. Moreover, I would like to thank my teacher and supervisor Dr. Siamak Khatibi and Blekinge Institute of Technology for giving me opportunity to work with a thesis that encompasses the knowledge of software and hardware as well as theories. My motivation to image processing and embedded development sharply increased. Further thanks to my father Md. Shofi Ullah, Mother Ayesha Akter for their continuous encouragement to move forward in search of knowledge. I am dedicating my thesis to my father and mother. 6 CONTENTS ABSTRACT ............................................................................................................................................ 3 ACKNOWLEDGEMENT ...................................................................................................................... 5 LIST OF FIGURES .............................................................................................................................. 15 LIST OF ACRONYMS ........................................................................................................................ 24 CHAPTER 1 ......................................................................................................................................... 26 INTRODUCTION ................................................................................................................................ 26 1.1 Thesis scope ................................................................................................................................ 29 1.2 Thesis Outline ............................................................................................................................. 33 CHAPTER 2 ......................................................................................................................................... 35 RESEARCH AND DEVELOPMENT METHODS ............................................................................. 35 CHAPTER 3 ......................................................................................................................................... 39 IMAGE STITCHING LITERATURE AND VISION SYSTEMS ....................................................... 39 3.1 Panoramic Image Stitching ......................................................................................................... 39 3.1.1 Photogrammetry ................................................................................................................... 40 3.1.2 Different Image stitching algorithms ................................................................................... 41 3.1.3 Image registration ................................................................................................................ 42 3.2 Real-time Panorama Vision Systems .......................................................................................... 47 3.2.1 MITRE immersive spherical vision system ......................................................................... 47 3.2.2 Point Grey Spherical Vision ................................................................................................ 48 3.2.3 Lucy S and Dot .................................................................................................................... 49 CHAPTER 4 ......................................................................................................................................... 51 EMBEDDED RTLINUX AND SOFTWARE DEVELOPMENT TOOLS ......................................... 51 4.1 Linux Kernel ............................................................................................................................... 51 4.2 Basics of Linux ........................................................................................................................... 52 7 4.3 Linux distributions ...................................................................................................................... 52 4.3.1 Ångström distribution .......................................................................................................... 53 4.3.2 Ubuntu Linux distribution .................................................................................................... 53 4.4 Embedded Linux ......................................................................................................................... 54 4.5 Real-time Linux .......................................................................................................................... 54 4.6 Bourne Shell ............................................................................................................................... 56 4.7 Text Editors ................................................................................................................................ 58 4.7.1 Vi editor ............................................................................................................................... 58 4.7.2 Nano editor ........................................................................................................................... 59 4.8 Native Compiler ......................................................................................................................... 60 4.9 OpenCV ...................................................................................................................................... 61 4.10 Native code and Linking Process ............................................................................................. 62 4.11 Build Automation Tool ............................................................................................................. 62 4.11.1 Build automation tool for OpenCV .................................................................................... 63 4.12 CMake ...................................................................................................................................... 63 4.12.1 Philosophy of using CMake ..............................................................................................