Provided by the author(s) and NUI Galway in accordance with publisher policies. Please cite the published version when available. Title Real-time detection of pedestrians in night-time conditions using a vehicle mounted infrared camera Author(s) Hurney, Patrick Publication Date 2016-09-30 Item record http://hdl.handle.net/10379/6374 Downloaded 2021-09-27T10:18:27Z Some rights reserved. For more information, please see the item record link above. Real-time Detection of Pedestrians in Night-time Conditions Using a Vehicle Mounted Infrared Camera A dissertation presented by Patrick Joseph Hurney to Electrical and Electronic Engineering College of Engineering and Informatics National University of Ireland, Galway in fulfillment of the requirements for the degree of Doctor of Philosophy in the subject of Electronic Engineering Supervisors Dr. Martin Glavin, Dr. Edward Jones and Dr. Fearghal Morgan Research Director Prof. W. G. Hurley September 2016 c 2016 - Patrick Joseph Hurney All rights reserved. Abstract Current statistics show that a significant number of road fatalities occur during night-time hours despite a smaller number of vehicles on the road. This number could be significantly reduced with the use of systems that automatically detect Vulnerable Road Users (VRUs) and alert the driver of their presence. This thesis presents an efficient embedded Advanced Driver Assistance System (ADAS) to detect pedestrians with a far-infrared sensor in real-time. The ADAS proposed in this thesis is implemented on a low power Intel Atom Central Processing Unit (CPU) and an Altera Arria GX II Field Programmable Gate Array (FPGA). The CPU and FPGA communicate over a PCI-express 2.0 bus using a Direct Memory Access engine on the FPGA. The pedestrian detection algorithm was partitioned between the FPGA and CPU to ensure real-time performance. Far-infrared images are first acquired with a microbolometer at a rate of 25fps, and a morphological closing operation is applied to remove distortion on the pedestrian's torso. The frames are sent to the FPGA to be processed using dedicated hardware. Regions of Interest (ROI) that could potentially contain pedestrians are isolated from the background using hardware accelerated Seeded Region Growing (SRG). The isolated ROI are classified using a Support Vector Machine (SVM) on the CPU. Histogram of Oriented Gradient (HOG) features and Local Binary Pattern (LBP) features are extracted from the ROI. These features are then concatenated to form a HOG-LBP feature vector and are passed to a classifier that determines if the ROI contains a pedestrian or non-pedestrian object. Successfully classified pedestrians are tracked between frames using a Kalman filter. The system runs in real-time at a rate of 25fps, the frame rate of the micro bolometer. The hardware accelerated SRG method obtained a 97.93% reduction in execution time compared to the software implementation than on the CPU alone. Detection rates of 98% have been achieved with HOG-LBP features on a database of 2,000 individual frames and video containing 15,000 frames. The total power consumption of the system is 3.12W. This results in a highly accurate, low power, low cost system suitable for the automotive sector. iii Declaration of Originality iv I hereby declare that the work contained in this thesis has not been submitted by me in pursuance of any other degree Signed: Date: Contents 1 Introduction 1 1.1 Motivation and Background . .1 1.2 Contributions of this Thesis . .4 1.3 Chapter by Chapter Overview . .5 2 Literature Review 7 2.1 Introduction . .7 2.2 Infrared Technology . .8 2.2.1 Infrared Radiation . .8 2.2.2 Infrared Sensors . .9 2.3 Overview of Pedestrian Detection Systems . 11 2.4 Region of Interest Isolation . 15 2.5 Region of Interest Classification . 20 2.6 Pedestrian Tracking . 23 2.7 Summary and Conclusions . 26 3 Embedded System Architecture 28 3.1 Introduction . 28 3.2 Processing Platforms . 28 3.2.1 Central Processing Unit (CPU) . 29 3.2.2 Digital Signal Processor (DSP) . 29 3.2.3 Graphics Processing Units (GPU) . 30 3.2.4 Field Programmable Gate Arrays (FPGA) . 31 3.3 Embedded ADAS . 32 3.3.1 CPU . 32 3.3.2 FPGA . 33 3.3.3 DMA Engine Device Driver . 34 3.4 System Testing . 34 3.5 Summary and Conclusion . 36 v Table of Contents vi 4 Region of Interest Isolation 38 4.1 Introduction . 38 4.2 Clothing Distortion Compensation . 39 4.3 Seeded Region Growing . 40 4.3.1 Binary Thresholding . 41 4.3.2 Connected Component Labelling . 41 4.3.3 Duplicate Removal . 46 4.4 Static Threshold vs Seeded Region Growing . 49 4.5 Software Execution Time . 50 4.6 Seeded Region Growing Hardware Design . 52 4.6.0.1 Two Pass CCL . 53 4.6.0.2 Multiple Pass CCL . 55 4.6.0.3 Parallel CCL . 56 4.6.0.4 Contour Tracing CCL . 56 4.6.0.5 Single Pass CCL . 58 4.6.1 Algorithm Timing Analysis . 62 4.7 SRG Hardware Architecture . 64 4.7.1 Seeded Region Growing Finite State Machine . 64 4.7.2 Binary Thresholding . 64 4.7.3 Connected Component Labelling . 66 4.7.3.1 Label Assignment . 66 4.7.3.2 Row Length Buffer . 69 4.7.3.3 Region Parameter Storage . 71 4.7.4 Region Filtering . 73 4.7.4.1 Aspect Ratio . 73 4.7.4.2 Location Based Filtering . 74 4.7.5 Duplicate Removal . 75 4.8 Resource Utilisation . 78 4.9 Hardware Accelerated SRG Execution Times . 78 4.10 Hardware Accelerated SRG Results . 79 4.11 Summary and Conclusion . 79 5 Region of Interest Classification 81 5.1 Introduction . 81 5.2 Region of Interest Feature Extraction . 82 5.2.1 Histogram of Oriented Gradients . 82 5.2.2 Local Binary Patterns . 83 5.2.3 Computation of LBP Feature Vector . 85 5.2.4 HOG-LBP Features . 87 5.2.5 Training Data . 88 5.2.6 Support Vector Machine Classifier . 89 5.3 Pedestrian Tracking . 89 Table of Contents vii 5.4 Classification Results . 92 5.4.1 Performance Metrics . 93 5.4.2 Classifier Performance . 94 5.4.3 Classifier Performance on FIR Video . 98 5.5 Execution Time . 101 5.6 Summary and Conclusion . 101 6 Conclusions and Future Work 104 6.1 Project Summary and Conclusions . 104 6.2 Primary Contributions . 107 6.3 Suggestions for Future Work . 108 Bibliography 110 A Seeded Region Growing Documentation 130 A.1 Introduction . 130 A.2 Binary Thresholding . 133 A.2.1 Data Dictionary . 133 A.3 Connected Component Labelling . 135 A.3.1 Label Assignment . 136 A.3.1.1 Data Dictionary . 136 A.3.2 Row Length Buffer . 139 A.3.2.1 Data Dictionary . 139 A.3.3 Region Parameter Storage . 141 A.3.3.1 Data Dictionary . 141 A.4 Duplicate Removal . 144 A.4.1 Data Dictionary . 144 B DMA Engine Overview 149 B.1 Linux device driver overview . 149 B.2 DMA Engine Parameters . 150 B.3 Transfer Procedure . 151 B.3.1 Initialise DMA engine . 151 B.3.2 Write Data to FPGA from Host . 152 B.3.3 Read Data from FPGA to Host . 152 B.3.4 Close DMA Engine . 152 B.4 DMA Descriptor Table . 153 B.5 DMA Engine Testing . 153 C Publications 154 C.1 Journals . 154 C.1.1 Published . 154 C.1.2 In Submission . 155 Table of Contents viii C.2 Conferences . 155 C.2.1 Published . 155 List of Figures 2.1 The electromagnetic spectrum . .8 2.2 Flowchart displaying the stages in a typical FIR pedestrian detection algorithm . 13 2.3 Images depicting various stages of FIR pedestrian detection algorithm 16 2.4 Static binary threshold operation performed on FIR image . 18 2.5 Examples of various templates used in probabilistic template matching. 21 3.1 Development prototype for the embedded pedestrian detection ADAS. The Intel N270 CPU is connected to the Altera Arria GX II FPGA over a PCI-express lane. 33 3.2 Linux kernel space . 35 4.1 Clothing distortion compensation . 40 4.2 Binary Thresholding process . 41 4.3 Connected Component Labelling process . 42 4.4 Forward and backward scanning masks . 43 4.5 CCL 4-way and 8-way masks applied to image . ..