Running head: RECOGNISING GESTURES USING TIME-SERIES ANALYSIS 1 RECOGNISING GESTURES USING TIME-SERIES ANALYSIS MICHAEL COOLEN TILBURG UNIVERSITY Student number: 2031591 Administration number: u741570 Email address: Supervisor: dr. P.A. Vogt Supervisor email address: Second reader: dr. M. Alimardani Course: 880502-M-18 (Master thesis/Data Science in Action) Faculty: Tilburg School of Humanities and Digital Sciences Department: Department of Cognitive Science and Artificial Intelligence Program: Data Science & Society Date: May 29th, 2019 Word count: 9.791 RECOGNISING GESTURES USING TIME-SERIES ANALYSIS 2 Preface This thesis has been written for the master program Data Science & Society at Tilburg University. Many thanks to my supervisor dr. Paul Vogt and Jan de Wit for providing me with this interesting research opportunity. Also, thanks to my fellow students for all the positive moments throughout the year. Finally, this thesis could not have been written without the support of my family and friends. RECOGNISING GESTURES USING TIME-SERIES ANALYSIS 3 Abstract Human-Robot Interaction is becoming more common. For robots to communicate in a natural way with humans, they will need to develop non-verbal communication. The first step in this process is to recognise human gestures. To recognise gestures, human gestures can be recorded using a motion device such as a Kinect. This research builds on previous research in which a large gesture dataset was created using Human-Robot interaction (de Wit et al., 2019a). In this previous study, in total 35 different types of gestures were recorded using a Kinect. Using one-shot learning, gestures were able to be classified with an accuracy of 23%. The goal of the current research was to find a method to increase this gesture classification accuracy. In this research, time-series analysis was used, which has not been done before in this research field. The dataset was transformed into a featureset with 4,960 features. Using feature selection, 146 features were found to be important. In total 23 machine learning algorithms were tested on the features. It was found that ensemble type algorithms work best for these kind of features. After hyperparameter tuning, it was found that a simple Random Forest was best in classifying gestures, with an accuracy of 47%. To increase this accuracy, three state-of-the-art ensemble algorithms were tested, which resulted in a classification accuracy of over 50% using CatBoost. For generalization purposes, a fast and simple model was created using the 15 most important time-series features. This model can achieve a classification accuracy of 35%. Keywords: Human-Robot Interaction, Gesture Recognition, Kinect, Time-series, Machine Learning RECOGNISING GESTURES USING TIME-SERIES ANALYSIS 4 Contents PREFACE .............................................................................................................................................................. 2 ABSTRACT .......................................................................................................................................................... 3 INTRODUCTION ............................................................................................................................................... 6 RELATED WORK ............................................................................................................................................... 9 RECORDING GESTURES ...................................................................................................................................... 9 ONE-SHOT LEARNING ...................................................................................................................................... 10 SUPERVISED LEARNING .................................................................................................................................... 11 TIME-SERIES ...................................................................................................................................................... 12 SUPERVISED LEARNING ALGORITHMS ............................................................................................................ 14 METHOD ........................................................................................................................................................... 17 SETUP ................................................................................................................................................................ 17 DATASET DESCRIPTION .................................................................................................................................... 17 PRE-PROCESSING .............................................................................................................................................. 18 FEATURE EXTRACTION ..................................................................................................................................... 20 FEATURESET PRE-PROCESSING ......................................................................................................................... 21 BASELINE .......................................................................................................................................................... 21 FEATURE SELECTION ........................................................................................................................................ 21 HYPERPARAMETER TUNING ............................................................................................................................ 23 VOTING ............................................................................................................................................................. 23 RESULTS ............................................................................................................................................................ 24 ALGORITHM COMPARISON .............................................................................................................................. 24 FEATURE SELECTION ........................................................................................................................................ 26 FIRST COMPARISON .......................................................................................................................................... 29 HYPERPARAMETER TUNING ............................................................................................................................ 31 VOTING ............................................................................................................................................................. 33 EXTRA ALGORITHMS ........................................................................................................................................ 34 GESTURES ......................................................................................................................................................... 36 DISCUSSION .................................................................................................................................................... 38 FEATURES ......................................................................................................................................................... 38 ALGORITHMS .................................................................................................................................................... 39 RECOGNISING GESTURES USING TIME-SERIES ANALYSIS 5 LIMITATIONS AND FUTURE RESEARCH ........................................................................................................... 41 CONCLUSION .................................................................................................................................................... 42 REFERENCES .................................................................................................................................................... 44 APPENDIX A ..................................................................................................................................................... 56 APPENDIX B ...................................................................................................................................................... 58 APPENDIX C ..................................................................................................................................................... 60 APPENDIX D ..................................................................................................................................................... 62 APPENDIX E ...................................................................................................................................................... 63 APPENDIX F ...................................................................................................................................................... 65 APPENDIX G ..................................................................................................................................................... 67 APPENDIX H ..................................................................................................................................................... 72 APPENDIX I ......................................................................................................................................................