• Abstract and Figures
• Public Full-text

WEARABLE SENSOR-BASED SIGN LANGUAGE RECOGNITION USING BIO-ACOUSTIC SIGNALS FROM HIGH-FREQUENCY ACCELEROMETERS by Karly N. Kudrinko A thesis submitted to the Department of Mechanical & Materials Engineering In conformity with the requirements for the degree of Master of Applied Science Queen’s University Kingston, Ontario, Canada December, 2020 Copyright ©Karly N. Kudrinko, 2020 Abstract Disabling hearing loss has a profound impact on an individual’s ability to interact and engage with those around them. Sign languages enable members of the Deaf community to communicate easily and effectively with one another. However, barriers exist for members of these populations when attempting to communicate with those who are not proficient in their language. The primary objective of this research was to develop a robust and user-oriented sign language recognition (SLR) system. This goal was accomplished through a comprehensive literature review of the wearable sensor-based SLR field, a national questionnaire to gather insights from members of the Deaf community, and the development of a novel bio-acoustic sensor system to recognize sign language gestures. Our comprehensive review of existing methods revealed many disadvantages of previous SLR systems making them unsuitable for practical applications. This literature review also revealed a disconnect between researchers and Deaf individuals, who are routinely noted as the potential end-users for SLR devices. Using this knowledge, a questionnaire was developed to gather insights and perspectives related to the SLR field directly from members of the Deaf community, family members and friends of Deaf individuals, and those who provide services targeted towards the Deaf community. Attributes including design specifications, potential applications, differences between American Sign Language and English, and other key considerations were revealed. Synthesizing our knowledge of the SLR field with the insights gathered from the questionnaire, we developed an unobtrusive wrist-worn SLR prototype. The wearable system is comprised of four high-frequency accelerometers capable of capturing mechanical micro-vibrations when placed over the major tendons of the wrist. Machine learning models were compared to demonstrate the feasibility of this system for the recognition of 15 American Sign Language gestures performed by multiple subjects. A series of validation experiments were carried out to demonstrate the effectiveness of the system. We also experimented with features to determine optimal sensor placement and the value of the high-frequency signals. Future iterations of this research will involve testing the system on larger lexicons of signs and adapting deep learning models to recognize continuous sign language sentences. ii Co-Authorship The contents of Chapters 2-4 of this thesis are multi-authored works described as follows: Chapter 2 contains the accepted version of a manuscript published in IEEE Reviews in Biomedical Engineering in its entirety. Minor modifications were made to the article when creating this chapter to maintain consistency. This work was supervised by Dr. Qingguo Li, who also provided constructive comments and edits. Emile Flavin assisted throughout the revision process. Dr. Xiaodan Zhu provided support related to the Recognition Models section of the paper and assisted with editing. My contributions included performing the comprehensive review of literature, developing figures and tables, and writing the bulk of the research article. This manuscript was reproduced in accordance with IEEE Copyright Policies. The full citation is as follows: © 2020 IEEE. Reprinted, with permission, from K. Kudrinko, E. Flavin, X. Zhu and Q. Li, “Wearable Sensor-Based Sign Language Recognition: A Comprehensive Review,” in IEEE Reviews in Biomedical Engineering. DOI: 10.1109/RBME.2020.3019769. In reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Queen’s University’s products or services. Internal or personal use of this material is permitted. If interested in reprinting/republishing IEEE copyrighted material for advertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/publications/rights/rights_link.html to learn how to obtain a License from RightsLink. If applicable, University Microfilms and/or ProQuest Library, or the Archives of Canada may supply single copies of the dissertation. Chapter 3 is an author’s original manuscript currently under review for publication in Assistive Technology. Since the manuscript has not been published, the copyright current remains with the authors. Article details are as follows: iii K. Kudrinko, E. Flavin, M. Shepertycky and Q. Li, “Assessing the Need for a Wearable Sign Language Recognition Device for Deaf Individuals: Results from a National Questionnaire.” Assistive Technology. Submitted: August 15, 2020. (Under review for publication) This work was supervised by Dr. Qingguo Li, who provided guidance and support throughout the development of both the questionnaire and manuscript. Emile Flavin assisted with formulating the questions included in the questionnaire, extracting common concerns from the responses, writing portions of the Abstract and Conclusions sections, and revision of the manuscript. Michael Shepertycky contributed through guidance related to statistical methods and revision of the manuscript. My contributions included developing the online questionnaire, distributing it to relevant organizations across Canada, performing the statistical analyses, and summarizing and interpreting findings in a research article. The contents of Chapter 3 are currently under preparation for publication. Dr. Qingguo Li supervised this work and provided many valuable suggestions throughout the development, testing, and analysis components of this research. The system prototype was constructed by Lei Wang (State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, China). Since this work is still under preparation for publication, the copyright remains with the authors. iv Acknowledgements I would like to express my sincerest gratitude toward everyone who has provided me with support and encouragement while I completed this thesis. A special thank you goes out to my research supervisor, Dr. Qingguo Li, for his unwavering guidance, passion, and positivity throughout my time in the Master of Applied Science program. I would like to thank my co-authors Dr. Xiaodan Zhu, Emile Flavin, Michael Shepertycky, and Lei Wang for their invaluable contributions. This research would not have been possible without their expertise, insights, and observations. I would also like to thank my lab mates for providing unique perspectives and meaningful conversations. I would like to thank my teammates and coaches of the varsity wrestling team for constantly pushing me beyond my limits. Finally, my deepest appreciation and gratitude goes out to my family members and friends. I would not be where I am today without their continuous support, love, and empowerment. v Table of Contents Abstract ......................................................................................................................................................... ii Co-Authorship.............................................................................................................................................. iii Acknowledgements ....................................................................................................................................... v List of Figures ............................................................................................................................................... x List of Tables .............................................................................................................................................. xii List of Abbreviations ................................................................................................................................. xiii Acronyms ............................................................................................................................................... xiii Chapter 1 Introduction .................................................................................................................................. 1 1.1 Motivation ........................................................................................................................................... 1 1.2 Research Objectives and Contributions .............................................................................................. 3 1.2.1 Wearable Sensor-Based Sign Language Recognition: A Comprehensive Review ...................... 3 1.2.2 Assessing the Need for a Wearable Sign Language Recognition Device for Deaf Individuals: Results from a National Questionnaire ................................................................................................. 4 1.2.3 A Bio-Acoustic Approach to American Sign Language Using Multiple Wrist-Worn High- Frequency Accelerometers .................................................................................................................... 4 1.3 Thesis Organization ............................................................................................................................ 4 Chapter 2 Wearable Sensor-Based

Load more

  129

Copy Link