School of Engineering and the Built Environment Hardware Architectures for Infrared Pedestrian Detection Systems by Robert Walczyk A thesis submitted in partial fulfilment of the requirements of Edinburgh Napier University, for the award of Doctor of Philosophy May 2013 Contact: [email protected] This document was set with the help of KOMA-Script and LATEX Abstract Infrared pedestrian detection systems struggle with real-time processing performance. Known solutions are limited to either low resolution systems with basic functionality running at full frame rate, or software based techniques featuring higher detection rates with full set of features, however running only in off-line mode for statistical analysis. Here, a comprehensive solution for real-time pedestrian detection is described. This research project includes investigation of possible solutions, design, development and implementation of a pedestrian detection system, processing data from infrared video source in real-time. Design requirements include processing at full frame rate as well as low memory and system resource consumption. The memory utilization is one of the major concerns since high demand for memory resources is a critical aspect in most image processing applications. For the purpose of this task, a number of general purpose image processing techniques were revised, taking into consideration the suitability for infrared pedestrian detection. These tasks include background separation, acquisition noise removal and object de- tection through connected component labelling. They are discussed and addressed in individual chapters. Various techniques for background segmentation are discussed. A chronological review of popular techniques is provided. The proposed architecture for background subtrac- tion is based on selective running average for adaptive background model, supported by adaptive thresholding based on histogram calculation. In order to remove acquisition noise, a dual decomposed architecture was introduced, based on mathematical morphol- ogy and basic set theory definitions. It includes both erosion and dilation performed in a pipeline. For the purpose of object detection and feature extraction, a connected component labelling technique was employed, based on a single pass approach to fulfil real-time processing requirement. The system was implemented, verified and tested on XUP FPGA Development Board with Virtex-II Pro XC2VP30 chip from Xilinx. Details and limitation of the specific implementation are discussed. An overview of experimental pedestrian detection results is provided. The thesis concludes with system analysis and suggestions for future work. iii Acknowledgements I would like to express my gratitude to all the people who have contributed towards this research or completion of this thesis. In particular, I would like to give thanks to the following: To my university supervisors Dr T. David Binnie and Dr Alistair Armitage. For your advice and continued encouragement throughout this research. I appreciate your feedback and value the time you have spent on reading and correcting this thesis. To my line manager from Cambridge Silicon Radio, Dr Tim Clapp, for generous study-leave allowance and flexible working hours during last months of my study. To Stefan Maagh for valuable advice and effort when creating this thesis template. To Piotr Wojtczuk for your contribution to this project in various aspects. Also, for our long night FPGA-whisky sessions, which I consider to be one of best experiences during my academic years. To Alexander Balazs, for your contribution while working on the implementation of tracking algorithms on embedded processor core. It was a great experience for me to act as a teacher and supervisor. I am glad this cooperation was concluded with joint research paper presented at the 2011 IET Irish Signals and Systems Conference (ISSC 2011) in Dublin. To my family, including my parents and sister in law - Krystyna, Janusz and Karolina Ko lodziejczak, for your continuous support. To my mother and father Lucyna and Waldemar, for your effort into my life-long education. I hope, after all these years, you can be proud of me. Special thanks to my wife Ania and our little baby girl Zosia. This research project has affected your life as much as mine. I believe only you know how much effort I have made towards the completion of this thesis. There is no words to express my gratitude for your support, understanding and patience. Robert Walczyk iv Acknowledgements v I acknowledge that some of the work presented in this thesis has been presented or published at or in the following places: [1] R. Walczyk, A. Balazs, A. Armitage, T.D. Binnie, "System architectures for infrared pedestrian tracking,"in: IET Irish Signals and Systems Conference (ISSC 2011), June 2011, Dublin, Ireland [2] R. Walczyk, A. Armitage, T.D. Binnie, "Comparative Study on Connected Component Labeling Algorithms for Embedded Video Processing Systems," in: Proceedings of the International Conference on Image Processing, Computer Vi- sion, and Pattern Recognition (IPCV'10), July 2010, Las Vegas, NV, USA [3] R. Walczyk, A. Armitage, T.D. Binnie, "FPGA Implementation of Hot Spot Detection in Infrared Video," in: IET Irish Signals and Systems Conference (ISSC 2010), June 2010, Cork, Ireland [4] R. Walczyk, A. Armitage, T.D. Binnie, "FPGA Implementation of Pedestrian Detection System using Infrared Image Sequences," in: Faculty of Engineering & Computing Postgraduate Research Conference (FECCI 2010), May 2010, Edin- burgh, UK [5] R. Walczyk, A. Armitage, T.D. Binnie, "An Embedded Real-Time Pedestrian Detection System Using an Infrared Camera," in: IET Irish Signals and Systems Conference (ISSC 2009), June 2009, Dublin, Ireland Contents Abstract iii Acknowledgements iv List of Figuresx List of Tables xiii List of Abbreviations xiv 1 Introduction 16 1.1 Motivation .................................. 17 1.2 Research Objective.............................. 18 1.3 Contribution to Knowledge ......................... 19 1.4 Outline of the Thesis............................. 19 2 Image Processing for Infrared Pedestrian Detection 21 2.1 Infrared Detectors .............................. 22 2.1.1 Infrared Radiation.......................... 22 2.1.2 Thermal Sensors........................... 23 2.1.3 Thermographic Cameras....................... 24 2.1.4 Applications ............................. 25 2.2 Digital Image Processing........................... 25 2.2.1 Digital Image............................. 26 2.2.2 Raster Scan.............................. 27 2.2.3 Pixel Neighbourhood......................... 27 2.2.4 Pixel Connectivity.......................... 28 2.2.5 Connected Components....................... 28 2.2.6 Scan Mask .............................. 29 2.2.7 Labelling ............................... 30 2.2.8 Run Length Encoding (RLE).................... 30 vi Contents vii 2.3 Hardware Development and Implementation................ 31 2.3.1 Processing Unit............................ 31 2.3.2 Design Flow.............................. 34 2.3.3 Development Platform........................ 36 2.4 Pedestrian Detection............................. 37 2.4.1 Segmentation............................. 38 2.4.2 Object detection........................... 39 2.4.3 Classification and Tracking ..................... 40 2.5 Conclusions.................................. 40 3 Background Segmentation 41 3.1 Introduction.................................. 41 3.2 Review of Background Subtraction Techniques .............. 43 3.2.1 Intensity Thresholding........................ 44 3.2.2 Temporal Difference......................... 46 3.2.3 Background Modelling........................ 48 3.3 Algorithm Overview ............................. 55 3.3.1 Background Modelling........................ 56 3.3.2 Histogram Calculation........................ 56 3.3.3 Feedback Loop Control ....................... 58 3.3.4 Algorithm Pseudo Code....................... 60 3.4 Hardware Implementation.......................... 61 3.4.1 Architecture ............................. 61 3.5 Conclusions.................................. 64 4 Morphology Filtering 67 4.1 Introduction.................................. 68 4.2 Mathematical Morphology.......................... 68 4.2.1 Set Theory .............................. 69 4.2.2 Structuring Element......................... 71 4.2.3 Erosion ................................ 72 4.2.4 Dilation................................ 72 4.2.5 Opening and Closing......................... 73 4.2.6 Duality ................................ 75 4.2.7 Decomposition ............................ 75 4.2.8 Summary............................... 76 4.3 Implementation................................ 77 4.3.1 Introduction ............................. 77 Contents viii 4.3.2 Previous Work ............................ 78 4.3.3 System Requirements ........................ 79 4.3.4 Architecture ............................. 80 4.3.5 Module Instantiation......................... 85 4.3.6 Results and Performance ...................... 86 4.4 Conclusions.................................. 91 5 Connected Component Labelling 94 5.1 Introduction.................................. 95 5.1.1 Labelling ............................... 96 5.1.2 Label Collision............................ 96 5.1.3 Feature Extraction.......................... 97 5.2 Literature Review .............................. 98 5.2.1 Labelling Algorithms......................... 98 5.2.2 Summary............................... 101 5.3 Algorithm