Multispectral Image Analysis for Object Recognition and Classification Claude Viau Thesis submitted to the Faculty of Graduate and Postdoctoral Studies in partial fulfillment of the requirements for the degree of Master of Applied Science in Electrical and Computer Engineering Ottawa-Carleton Institute for Electrical and Computer Engineering School of Electrical Engineering and Computer Science University of Ottawa March 2016 © Claude Viau, Ottawa, Canada, 2016 ii Abstract Computer and machine vision applications are used in numerous fields to analyze static and dynamic imagery in order to assist or automate some form of decision-making process. Advancements in sensor technologies now make it possible to capture and visualize imagery at various wavelengths (or bands) of the electromagnetic spectrum. Multispectral imaging has countless applications in various field including (but not limited to) security, defense, space, medical, manufacturing and archeology. The development of advanced algorithms to process and extract salient information from the imagery is a critical component of the overall system performance. The fundamental objectives of this research project were to investigate the benefits of combining imagery from the visual and thermal bands of the electromagnetic spectrum to improve the recognition rates and accuracy of commonly found objects in an office setting. The goal was not to find a new way to “fuse” the visual and thermal images together but rather establish a methodology to extract multispectral descriptors in order to improve a machine vision system’s ability to recognize specific classes of objects. A multispectral dataset (visual and thermal) was captured and features from the visual and thermal images were extracted and used to train support vector machine (SVM) classifiers. The SVM’s class prediction ability was evaluated separately on the visual, thermal and multispectral testing datasets. Commonly used performance metrics were applied to assess the sensitivity, specificity and accuracy of each classifier. The research demonstrated that the highest recognition rate was achieved by an expert system (multiple classifiers) that combined the expertise of the visual-only classifier, the thermal-only classifier and the combined visual-thermal classifier. iii Acknowledgment I would like to offer my sincere gratitude to Dr. Pierre Payeur and Dr. Ana-Maria Cretu for all your support with this research project. You have provided invaluable insight, subject matter expertise and guidance along the way. To my wife and daughters, many sacrifices were made on this journey and I am truly grateful for your patience and unconditional support. iv Table of Contents ABSTRACT ...................................................................................................................................................... II ACKNOWLEDGMENT .................................................................................................................................... III LIST OF FIGURES ........................................................................................................................................... VI LIST OF TABLES ........................................................................................................................................... VIII LIST OF ACRONYMS ...................................................................................................................................... IX CHAPTER 1 INTRODUCTION ..................................................................................................................... 1 1.1. CONTEXT ........................................................................................................................................... 1 1.2. RESEARCH OBJECTIVES ..................................................................................................................... 2 1.3. THESIS ORGANIZATION ..................................................................................................................... 3 CHAPTER 2 LITERATURE REVIEW ............................................................................................................. 4 2.1. SEGMENTATION ................................................................................................................................ 4 2.2. IMAGE FEATURES .............................................................................................................................. 8 2.3. CLASSIFIERS ..................................................................................................................................... 14 2.4. PERFORMANCE METRICS ................................................................................................................ 16 2.5. SUMMARY OF LITERATURE REVIEW ............................................................................................... 18 CHAPTER 3 DATA COLLECTION AND DATASETS ..................................................................................... 20 3.1. CAMERA SPECIFICATIONS AND IMAGE ANALYSIS SOFTWARE ....................................................... 20 3.2. IMAGE PREPROCESSING.................................................................................................................. 24 CHAPTER 4 METHODOLOGY .................................................................................................................. 31 4.1. OVERVIEW ....................................................................................................................................... 31 4.2. SEGMENTATION .............................................................................................................................. 31 4.2.1. BASIC THRESHOLD....................................................................................................................... 32 4.2.2. K-MEANS ..................................................................................................................................... 32 4.2.3. CONTOURS .................................................................................................................................. 32 4.2.4. WATERSHED WITH DISTANCE TRANSFORM ............................................................................... 32 4.2.5. PERFORMANCE ASSESSMENT ..................................................................................................... 33 4.2.6. WATERSHED WITH THERMAL MARKERS .................................................................................... 37 4.3. IMAGE FEATURE SELECTION ........................................................................................................... 41 4.4. CLASSIFIER SELECTION .................................................................................................................... 47 CHAPTER 5 SOFTWARE IMPLEMENTATION AND DATA PROCESSING ................................................... 48 v 5.1. OVERVIEW ....................................................................................................................................... 48 5.2. SEGMENTATION .............................................................................................................................. 50 5.3. FEATURE EXTRACTION AND POST PROCESSING ............................................................................. 52 5.4. TRAINING AND TESTING THE CLASSIFIERS ...................................................................................... 56 CHAPTER 6 EXPERIMENTAL EVALUATION ............................................................................................. 57 6.1. EXPERIMENT DESIGN ...................................................................................................................... 57 6.2. CLASSIFICATION RESULTS ............................................................................................................... 59 6.2.1. INDIVIDUAL FEATURES ................................................................................................................ 59 6.2.2. VISUAL BAND .............................................................................................................................. 64 6.2.3. THERMAL BAND .......................................................................................................................... 67 6.2.4. COMBINED VISUAL AND THERMAL BANDS ................................................................................ 71 6.2.5. EXPERT SYSTEM ........................................................................................................................... 76 6.2.6. FURTHER IMPROVEMENTS ......................................................................................................... 78 CHAPTER 7 CONCLUSIONS ..................................................................................................................... 80 7.1. SUMMARY OF FINDINGS ................................................................................................................. 80 7.2. CONTRIBUTIONS ............................................................................................................................. 81 7.3. FUTURE WORK ................................................................................................................................ 82 REFERENCES ................................................................................................................................................ 84 vi List of Figures