Simultaneous Adaptive Fractional Discriminant Analysis: Applications to the Face Recognition Problem Dissertation Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy in the Graduate School of The Ohio State University By John Daniel Draper, B.S., M.S. Graduate Program in Statistics The Ohio State University 2012 Dissertation Committee: Dr. Prem Goel, Advisor Dr. Radu Herbei Dr. Yoonkyung Lee c Copyright by John Daniel Draper 2012 Abstract Linear Discriminant Analysis (LDA) has served as a standard technique in classifica- tion for many years. Many improvements have been proposed to boost the performance of LDA yet maintain its simplicity, intuitive appeal, and robust nature. Lotlikar and Kothari proposed fractional LDA (F-LDA) as an improved version of LDA. However, for a large number of classes, F-LDA is not feasible to implement in real time due to huge computa- tional effort in the sequential search process for each dimension to be reduced. In addition, F-LDA is a directed projection pursuit technique, which takes several iterations to reduce just one dimension. Our research is focused on modifying these methods to be applicable to the face recognition problem (high-dimensional image data, large number of classes). Simultaneous Adaptive Fractional Discriminant Analysis (SAFDA) is a procedure devel- oped specifically to learn a specified or fixed low-dimensional subspace in which classes are well separated by sequentially downweighting all dimensions to be removed simulta- neously. Via analysis of a weighted between class scatter matrix of whitened data, S˜B, the best projected space is learned through a directed projection pursuit method that focuses on class separation in the reduced space (rather than the full space like LDA). An adaptive ker- nel (Gaussian) was found to be the most suitable to avoid extra time considerations inherent in cross-validation by allowing the data to determine optimal bandwidth choice. While the SAFDA algorithm showed a marked improvement over standard LDA techniques in terms ii of classification, the additional computational time, compared to LDA, was minimal in sit- uations involving a small number of classes (MNIST) as well as a large number of classes (AR face database). SAFDA also provides a procedure that matches (or in some cases, outperforms) the F-LDA benchmark in terms of classification, yet is much more feasible in computational time and effort. iii This work is dedicated to my family for their constant support throughout my life–my grandfather, Alfred C. Draper, for showing me the wonders of numbers and mathematics; my sister, Allison G. Draper, for always keeping me grounded; my mother, Karen C. Zawada, for her ceaseless love and support; and my father, Daniel V. Draper, for his constant thoughts, prayers, advice, and late night conversations that kept me going whenever things seemed overwhelming. Thank you all for everything! iv Acknowledgments This work would never have been possible without the support of countless individuals and organizations. My deepest thanks go out to: • My advisor, Dr. Prem Goel, for sticking with me through the years and providing countless hours of advice and statistical knowledge. • My committee, Dr. Yoonkyung Lee and Dr. Radu Herbei, for their patience and insight. • The OSU Department of Statistics, for providing countless teaching opportunities and allowing me to find my niche. • Dr. Jon R. Woods and The Ohio State University Marching Band (TBDBITL), for providing me with an organization that has made my college years a joy. I only hope that I can say that I left TBDBITL a little better than I found it. • And finally, and most importantly, to my family, for their countless thoughts, prayers, and support throughout the years. I could have never reached this milestone without the immense support and advice I have received from everyone. I can never repay the debt of gratitude that I owe to these in- dividuals, but I will continue to strive for the highest and, as Woody said, ‘Pay it Forward’. v Vita August 23, 1981 . Born - Dayton, OH 2003 . B.S. Mathematics–Florida State Univer- sity 2003 . B.S. Statistics–Florida State University 2005 . M.S. Statistics–The Ohio State University 2004-present . .Graduate Teaching Associate/Lecturer, The Ohio State Univeristy. 2011 . Interdisciplinary Specialization in Com- prehensive Engineering and Science of Biomedical Imaging–The Ohio State University Fields of Study Major Field: Statistics vi Table of Contents Page Abstract......................................... ii Dedication........................................ iv Acknowledgments....................................v Vita........................................... vi List of Tables...................................... ix List of Figures......................................x 1. Introduction and Literature Review........................1 1.1 Dimension Reduction, Feature Selection, and Classification.......2 1.1.1 Why not use all the data?.....................2 1.1.2 Dimensions vs. Features: Is there a difference?..........5 1.1.3 Dimension Reduction Techniques.................8 1.1.4 Classification Techniques..................... 30 1.2 Face Recognition Problem......................... 41 1.2.1 What Aspects are Central to Human Cognition of Faces?.... 43 1.2.2 Dimension Reduction for Faces.................. 44 1.2.3 Unifying Aspects of Face Recognition.............. 52 2. Simultaneous Adaptive Fractional Discriminant Analysis............ 53 2.1 Motivation and Notation.......................... 53 2.1.1 Notation.............................. 55 2.2 Why Weight?................................ 56 2.3 Description and Algorithm......................... 58 2.3.1 Downweighting Schemes..................... 65 vii 2.4 Connections between SAFDA and Projection Pursuit........... 68 2.4.1 Simple Example.......................... 71 3. Case Studies, Performance, and Implementation Issues............. 75 3.1 Introduction................................. 75 3.2 Case Study Databases........................... 77 3.3 Choice of Weighting Function and Parameter Selection.......... 79 3.3.1 Inverse Pairwise Distance Similarity Measure.......... 82 3.3.2 Bounded Inverse Pairwise Distance Similarity Measure..... 99 3.3.3 Gaussian Kernel.......................... 106 3.4 Stopping Criterion............................. 117 3.4.1 Case Study 1: MNIST database.................. 119 3.4.2 Case Study 2: AR database.................... 122 3.5 Downweighting Schemes......................... 125 3.6 Performance and Computation Time Comparison for Different Algorithms127 3.6.1 Case Study 1: MNIST database with 10 classes.......... 127 3.6.2 Case Study 2: AR Database with 100 classes........... 129 3.6.3 Configurations of Projected Centroids through the Fractional Shrink- ing Iterations and Confusion Matrices............... 133 3.6.4 Discussion............................. 143 4. Contributions and Future Work.......................... 145 4.1 Discussion and Conclusion......................... 145 4.2 Future Work................................ 148 Bibliography 153 viii List of Tables Table Page 2.1 LDA Confusion Matrix: Simulated Data................... 72 2.2 SAFDA Confusion Matrix: Simulated Data................. 73 3.1 CCR and Time for SAFDA Algorithms under Different Downweighting Schemes: AR Database............................ 126 3.2 CCR for MNIST database for Different Procedures using Gaussian Kernel Weight Function................................ 127 3.3 Time (sec) for MNIST database for Different Procedures using Gaussian Kernel Weight Function............................ 128 3.4 CCR for 10 random splits of AR database for Different Procedures using Bounded Inverse Pairwise Distance Weighting Function with coarse CV.. 130 3.5 Time (sec) for 10 random splits of AR database for Different Procedures using Bounded Inverse Pairwise Distance Weighting Function with coarse CV....................................... 130 3.6 CCR for 10 random splits of AR database for Different Procedures using Gaussian Kernel................................ 131 3.7 Time (sec) for 10 random splits of AR database for Different Procedures using Gaussian Kernel............................ 132 3.8 MNIST Confusion Matrix under Optimal LDA for d=4........... 137 3.9 MNIST Confusion Matrix under SAFDA for d=4.............. 137 ix List of Figures Figure Page 2.1 One-Dimensional Projection of Test Data in Classes 1 and 2 under LDA vs. SAFDA.................................. 74 3.1 MNIST examples............................... 78 3.2 AR Example Images............................. 79 3.3 Effects of Parameter h on CCR Inverse Pairwise Distance Measure for MNIST d=4 (Training/Validation/Test Split)................. 84 3.4 Eigenvalue Separation for MNIST data using Inverse Pairwise Distance Measure; h=19.9, d=4............................. 86 3.5 Progression of Weights using Inverse Pairwise Distance Measure for MNIST data; h=19.9, d=4............................... 87 3.6 CCR Progression Inverse Pairwise Distance Measure for MNIST data; h=19.9, d=4...................................... 88 3.7 Eigenvalue Separation for MNIST data using Inverse Pairwise Distance Measure; h=7.1, d=4............................. 90 3.8 Progression of Weights using Inverse Pairwise Distance Measure for MNIST data; h=7.1, d=4................................ 91 3.9 CCR Progression Inverse Pairwise Distance Measure for MNIST data; h=7.1, d=4...................................... 91 3.10 Effects of Parameter h on