IMAGE CLASSIFICATION USING TRANSFER LEARNING AND CONVOLUTION NEURAL NETWORKS A Paper Submitted to the Graduate Faculty of the North Dakota State University of Agriculture and Applied Science By Mohan Burugupalli In Partial Fulfillment of the Requirements for the Degree of MASTER OF SCIENCE Major Department: Computer Science July 2020 Fargo, North Dakota North Dakota State University Graduate School Title IMAGE CLASSIFICATION USING TRANSFER LEARNING AND CONVOLUTION NEURAL NETWORKS By Mohan Burugupalli The Supervisory Committee certifies that this disquisition complies with North Dakota State University’s regulations and meets the accepted standards for the degree of MASTER OF SCIENCE SUPERVISORY COMMITTEE: Dr. Simone Ludwig Chair Dr. Saeed Salem Dr. Maria Alfonseca Cubero Approved: July 30, 2020 Dr. Kendall Nygard Date Department Chair ABSTRACT In the recent years, deep learning has shown to have a formidable impact on image classification and has bolstered the advances in machine learning research. The scope of image recognition is going to bring big changes in the Information Technology domain. This paper aims to classify medical images by leveraging the advantages of Transfer Learning over Conventional methods. Three types of approaches are used namely, pre-trained CNN as a Feature Extractor, Feature Extractor with Image Augmentation, and Fine-tuning with Image Augmentation. The best pre-trained network architectures such as VGG16, VGG19, ResNet50, Inception, Xception and DenseNet are used for classification with each being applied to all the three approaches mentioned. The results are captured to find the best combination of pre-trained network and an approach that classifies the medical datasets with a higher accuracy. iii ACKNOWLEDGMENTS I would like to thank everyone who inspired me to work on the “Image Classification project”. First and foremost, I thank my adviser, Professor Simone Ludwig for introducing me to the concept of “Convolution Neural Networks” through her course “Advanced Intelligent Systems”. Also, I always admire her for the quick and prompt responses, providing me all the encouragement and support needed for completing this paperwork successfully. I would also like to thank my committee members Professor Saeed Salem, Professor Maria Alfonseca Cubero for their interest in my work and willingness to serve on my committee. Finally, I would like to thank my parents and friends for their unending support and inspiration. iv TABLE OF CONTENTS ABSTRACT ................................................................................................................................... iii ACKNOWLEDGMENTS ............................................................................................................. iv LIST OF TABLES ....................................................................................................................... viii LIST OF FIGURES ....................................................................................................................... ix 1. INTRODUCTION ...................................................................................................................... 1 1.1. Understanding of a Neural Network..................................................................................... 1 1.1.1. Basic Building Block of Neural Networks – “Neuron” ................................................ 1 1.1.2. Neural Network ............................................................................................................. 2 1.2. Composition of a Convolution Neural Network .................................................................. 2 1.3. Transfer Learning ................................................................................................................. 3 2. RELATED WORK ..................................................................................................................... 5 3. APPROACH ............................................................................................................................... 7 3.1. How a Computer See an Image ............................................................................................ 7 3.1.1. Colored Images .............................................................................................................. 7 3.1.2. Gray Scale Images ......................................................................................................... 8 3.2. Convolution Neural Networks.............................................................................................. 9 3.2.1. Assigning Values According to Color ........................................................................... 9 3.2.2. Selecting Features for Convolution ............................................................................. 10 3.2.3. Convolution Layer ....................................................................................................... 10 3.2.4. ReLU Layer ................................................................................................................. 12 3.2.5. Pooling Layer .............................................................................................................. 13 3.2.6. Stacking Up the Layers................................................................................................ 14 3.2.7. Fully Connected Layer ................................................................................................ 15 3.2.8. Output .......................................................................................................................... 15 v 3.3. Transfer Learning ............................................................................................................... 17 3.3.1. Why to Use Transfer Learning .................................................................................... 17 3.4. Architectures of Pre-trained Models .................................................................................. 20 3.4.1. VGG16 Architecture ................................................................................................... 20 3.4.2. VGG19 Architecture ................................................................................................... 21 3.4.3. ResNet50 Architecture ................................................................................................ 21 3.5. Leveraging Transfer Learning with Pre-trained CNN Models........................................... 22 3.5.1. Feature Extractor ......................................................................................................... 22 3.5.2. Feature Extractor with Image Augmentation .............................................................. 22 3.5.3. Fine-Tuning with Image Augmentation ....................................................................... 22 4. EXPERIMENTS AND RESULTS ........................................................................................... 24 4.1. Data Composition ............................................................................................................... 24 4.1.1. Initial Datasets ............................................................................................................. 24 4.1.2. Labelled and Organized Datasets ................................................................................ 24 4.2. Performance Metrics Setup ................................................................................................ 26 4.2.1. Accuracy and Precision ............................................................................................... 26 4.2.2. Recall/Sensitivity and Specificity/Selectivity ............................................................. 27 4.2.3. F1-Score ...................................................................................................................... 27 4.2.4. Confusion Matrix......................................................................................................... 28 4.2.5. ROC Curve .................................................................................................................. 28 4.3. Three Methodologies of Classification .............................................................................. 29 4.3.1. Pre-trained CNN Model as Feature Extractor ............................................................. 29 4.3.2. Pre-trained CNN Model as a Feature Extractor with Image Augmentation ................ 30 4.3.3. Pre-trained Model with Fine-Tuning and Image Augmentation ................................. 31 vi 4.4. Evaluation Results .............................................................................................................. 33 4.4.1. Random Test Images Classification ............................................................................ 33 4.4.2. Accuracy ...................................................................................................................... 34 4.4.3. Precision ...................................................................................................................... 35 4.4.4. Recall/Sensitivity ......................................................................................................... 36 4.4.5. Specificity .................................................................................................................... 37 4.4.6. F1-score ......................................................................................................................