UNIVERSITY OF THE WESTERN CAPE Robust South African Sign Language gesture recognition using hand motion and shape by Ibraheem Frieslaar A thesis submitted in fulfillment for the degree of Master of Science in the Faculty of Science Department of Computer Science Supervisor: Mehrdad Ghaziasgar Co-supervisor: James Connan February 2014 Declaration I, Ibraheem Frieslaar, declare that this thesis “Robust South African Sign Language gesture recognition using hand motion and shape” is my own work, that it has not been submitted before for any degree or assessment at any other university, and that all the sources I have used or quoted have been indicated and acknowledged by means of com- plete references. Signature: . Date: ........................ i “Seek knowledge from the cradle to the grave.” Abstract Research has shown that five fundamental parameters are required to recognize any sign language gesture: hand shape, hand motion, hand location, hand orientation and facial expressions. The South African Sign Language (SASL) research group at the University of the Western Cape (UWC) has created several systems to recognize sign language gestures using single parameters. These systems are, however, limited to a vocabulary size of 20 – 23 signs, beyond which the recognition accuracy is expected to decrease. The first aim of this research is to investigate the use of two parameters – hand motion and hand shape – to recognise a larger vocabulary of SASL gestures at a high accuracy. Also, the majority of related work in the field of sign language gesture recognition using these two parameters makes use of Hidden Markov Models (HMMs) to classify gestures. Hidden Markov Support Vector Machines (HM-SVMs) are a relatively new technique that make use of Support Vector Machines (SVMs) to simulate the functions of HMMs. Research indicates that HM-SVMs may perform better than HMMs in some applications. To our knowledge, they have not been applied to the field of sign language gesture recognition. This research compares the use of these two techniques in the context of SASL gesture recognition. The results indicate that, using two parameters results in a 15% increase in accuracy over the use of a single parameter. Also, it is shown that HM-SVMs are a more accurate technique than HMMs, generally performing better or at least as good as HMMs. Keywords Hidden Markov Models, Support Vector Machines, Hidden Markov Support Vector Ma- chine, Face Detection, Skin Detection, Background Subtraction, Hand Shape Recogni- tion, Hand Motion iii Acknowledgements First and foremost, all praise is given to Allah, the most merciful and most gracious. Thank you for giving me a healthy body and allowing me the ability to seek knowledge. I would like to thank my parents for ensuring that I receive a proper education and having the patience to allow me to further my studies. My sincerest appreciation is given to Mr James Connan for having given me the oppor- tunity to work for him or, as he would say, pretend to work. If it wasn’t for you Sir, I would probably be standing in some clothing store wasting my life away. I would like to thank Mr Mehrdad Ghaziasgar for reminding me that we are value adders and not value seekers. Thank you for always pushing me to not achieve the best, but to strive for something greater. It has been a pleasure to have you as a supervisor and receive your words of wisdom. A personal thank you goes to Dane Brown for all the years of friendship. You are not just a friend, but I see you as a brother. Thank you for the endless advice, even though it seems I never listen to you, and thank you for putting up with all my drama and problems. To my brother from another mother, Gary Jansen, thank you for reminding me that there is life out there, and for all the crazy and epic moments in the Jungle. I would also like to thank my lab mates Jihad Mohammad, Warren Nel, Diego Mush- fieldt, Kenzo Abrahams, Imran Achmed and Roland Foster for all the moments in the lab, especially being trolled every day. It has been an honour serving with you guys. Contents Declaration of Authorship i Abstract iii Keywords iii Acknowledgements iv List of Figures viii List of Tables x Abbreviations xi 1 Introduction 1 1.1 Background and Motivation .......................... 1 1.2 Research Question ............................... 4 1.3 Research Objectives .............................. 4 1.4 Premises ..................................... 5 1.5 Thesis Outline ................................. 5 2 Related Work 7 2.1 Articulated Object Recognition ........................ 7 2.1.1 Template Matching Techniques .................... 8 2.1.2 Machine Learning Techniques ..................... 12 2.1.3 Summary of Articulated Object Recognition . 15 2.2 Gesture Recognition using Hidden Markov Models . 15 2.2.1 Glove-Based Systems .......................... 16 2.2.2 Hardware-Based Systems ....................... 18 2.2.3 Vision-Based Systems ......................... 19 2.3 Hidden Markov Support Vector Machines (HM-SVMs) . 21 2.4 Summary of the Related Work ........................ 23 3 Image Processing in Gesture Recognition 24 v Contents vi 3.1 Face Detection ................................. 24 3.1.1 Haar-like Wavelet Detection ...................... 25 3.1.2 Integral Image ............................. 26 3.1.3 AdaBoost ................................ 26 3.1.4 Construction of a Rejection Cascade of Weak Classifier Nodes . 27 3.1.5 Testing and Results on the Face Detection Method . 28 3.2 Skin Detection ................................. 28 3.2.1 RGB Colour Space ........................... 29 3.2.2 Normalised RGB Colour Space .................... 29 3.2.3 HSV Colour Space ........................... 30 3.2.4 YCbCr colour space ......................... 31 3.2.5 Selecting An Appropriate Colour Space for Skin Detection . 32 3.2.6 Skin Model ............................... 33 3.3 Background Subtraction ............................ 34 3.3.1 Static Background Subtraction .................... 35 3.3.2 Frame Differencing ........................... 35 3.3.3 Statistical Background Subtraction . 36 3.3.4 A Comparison of the Background Subtraction Techniques . 37 3.4 Image Smoothing Using Gaussian Blur .................... 38 3.5 Feature Normalisation by Applying a Minimum Bounding Rectangle . 40 3.6 Summary and Conclusion ........................... 41 4 Support Vector Machines, Hidden Markov Models and a Hybrid Ap- proach 43 4.1 Support Vector Machines ........................... 43 4.1.1 Kernel Functions ............................ 48 4.1.2 SVM Multi-Class Classification .................... 49 4.1.2.1 One-vs-Others ........................ 49 4.1.2.2 One-vs-One .......................... 50 4.1.2.3 Directed Acyclic Graph (DAG) Technique . 50 4.2 Hidden Markov Models (HMMs) ....................... 51 4.2.1 Markov Chain ............................. 51 4.2.2 Formulation of HMMs Based on Markov Models . 54 4.2.3 The Three Basic Problems for HMMs . 56 4.2.3.1 A Solution to the Evaluation Problem . 57 4.2.3.2 A Solution to the Decoding Problem . 57 4.2.3.3 A Solution to the Learning Problem . 59 4.3 Hidden Markov Support Vector Machines (HM-SVMs) . 60 4.3.1 Input and Output Mappings via Joint Feature Functions . 60 4.3.2 Hidden Markov Chain Discriminants . 62 4.3.3 Hidden Markov Perceptron Learning . 63 4.3.4 Hidden Markov Support Vector Machine . 64 4.4 Summary .................................... 65 5 Design and Implementation of the Gesture Recognition System 66 5.1 Feature Extraction ............................... 66 5.1.1 Skin Detection ............................. 68 Contents vii 5.1.2 The Locating Procedure ........................ 69 5.1.3 The Correction Procedure ....................... 71 5.1.4 Hand Shape Recognition ........................ 71 5.1.5 Hand Motion Extraction ........................ 73 5.2 Feature Vector ................................. 74 5.3 Conclusion ................................... 75 6 Experimental Results and Analysis 76 6.1 Experimental Setup .............................. 77 6.2 Testing of the Hand Tracking Component . 77 6.2.1 Hand Tracking Experimental Procedure . 77 6.2.2 Results and Analysis .......................... 78 6.3 Training and Testing of the Hand Shape Recognition Component . 79 6.3.1 Training Set ............................... 79 6.3.2 Training and Optimisation of the Radial Basis Function Kernel Parameters ............................... 80 6.3.3 Hand Shape Recognition Experimental Procedure . 81 6.3.4 Results and Analysis .......................... 82 6.4 Testing of the Proposed SASL Gesture Recognition System . 83 6.4.1 Data Set ................................. 84 6.4.2 Feature Vector Comparison Experiment . 86 6.4.2.1 Feature Vector Comparison Experimental Procedure . 86 6.4.2.2 Results and Analysis .................... 87 6.4.3 HMM versus HM-SVM Comparison Experiment . 90 6.4.3.1 HMM versus HM-SVM Comparison Experimental Pro- cedure ............................. 90 6.4.3.2 Results and Analysis .................... 91 6.5 Conclusion ................................... 94 7 Conclusion 96 7.1 Directions for Future Work .......................... 97 7.1.1 Optimising HM-SVMs ......................... 97 7.1.2 Parallel HMMs ............................. 97 7.1.3 Including Additional Sign Language Parameters . 97 7.2 Concluding Remarks .............................. 98 A Additional Test Results 99 Bibliography 102 List of Figures 2.1 The skin image of the hand with the green boxes representing the matched features. [99]. .................................