REAL TIME IMPLEMENTATION OF DIRECTION OF ARRIVAL ESTIMATION ON ANDROID PLATFORMS FOR HEARING AID APPLICATIONS by Abdullah Küçük APPROVED BY SUPERVISORY COMMITTEE: ___________________________________________ Dr. Issa M. S. Panahi, Chair ___________________________________________ Dr. Carlos Busso ___________________________________________ Dr. Mehrdad Nourani Copyright 2018 Abdullah Küçük All Rights Reserved To my wife, family and every person who had faith in me. REAL TIME IMPLEMENTATION OF DIRECTION OF ARRIVAL ESTIMATION ON ANDROID PLATFORMS FOR HEARING AID APPLICATIONS by Abdullah Küçük, BE THESIS Presented to the Faculty of The University of Texas at Dallas in Partial Fulfillment of the Requirements for the Degree of MASTER OF SCIENCE IN ELECTRICAL ENGINEERING THE UNIVERSITY OF TEXAS AT DALLAS AUGUST 2018 ACKNOWLEDGMENTS This thesis is the result of two years of research regarding speech source localization. I am grateful to finish my thesis thanks to valuable feedback from many people. I am thankful to all who encouraged and trusted my aspirations and endeavors. I would like to express my gratitude to my advisor, Dr. Issa Panahi for giving me the opportunity to work on this project at the Statistical Signal Processing Laboratory (SSPRL) / UT Acoustic Laboratory (UTAL) and for his valuable mentorship and guidance throughout my research. I also am thankful to Dr. Carlos Busso and Dr. Mehrdad Nourani for serving as committee members for my master thesis defense. I would like to thank my lab members for helping my research: Mr. Anshuman (for supervising me on my research topic), Mr. Yiya Hao (for helping me with real-time implementation of the algorithm), Mr. Chandan Reddy (for his help on DSP related doubts), Mr. Gautam Bhat, Mr. Nikhil Shankar, Mr. Parth Mishra, Mr. Serkan Tokgoz, Ms. Ziyan Zou (for useful discussions about our research and empathizing with difficulties of research). I am thankful to my wife, my parents, my parents-in-law, my siblings, my siblings-in-law, and my friends for being understanding and their moral support. Thank you for trusting and encouraging me. This thesis was supported by the National Institute of the Deafness and Other Communication Disorders (NIDCD) of the National Institutes of Health (NIH) under Award 1R01DC015430-01. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. April 2018 v REAL TIME IMPLEMENTATION OF DIRECTION OF ARRIVAL ESTIMATION ON ANDROID PLATFORMS FOR HEARING AID APPLICATIONS Abdullah Küçük, MSEE The University of Texas at Dallas, 2018 ABSTRACT Supervising Professor: Dr. Issa M.S. Panahi Sound Source Localization (SSL) is one of the vital areas in signal processing, especially in hearing aid applications. SSL (or Direction of Arrival) helps to determine the location of the speaker via multiple fixed microphones (also known as microphone array). Knowing of speaker Direction of Arrival (DOA) helps to improve the performance of the system. Another advantage of DOA is that it helps hearing-impaired people to locate talker because hearing aid users, especially those who are older than 60 years, have difficulties in determining the place of the speaker. Having requisite processing capabilities and at least two microphones makes smartphones a cost-effective solution for multi-channel audio signal processing. In this thesis, we propose a new stereo input/output framework for Android platforms for audio signal processing. This frame enables us to perform multi or single channel audio signal processing for real-time operations. We also propose a method for two microphones-based Direction of Arrival (DOA) estimation and real-time implementation of this method on the latest Android smartphones. vi TABLE OF CONTENTS ACKNOWLEDGEMENTS .............................................................................................................v ABSTRACT ................................................................................................................................... vi LIST OF FIGURES ....................................................................................................................... ix LIST OF TABLES ......................................................................................................................... xi CHAPTER 1 INTRODUCTION ...................................................................................................1 1.1. Motivation ................................................................................................................1 1.2. Contribution of the thesis .........................................................................................2 1.2.1 Stereo Channel Input and Output Framework for Android Platforms .........2 1.2.2 Speech Source Localization and Tracking ...................................................3 1.2.3 Real Time Implementation of Proposed DOA Estimation Method on Android Smartphone ................................................................................................4 1.3. Outline of the Thesis ................................................................................................5 CHAPTER 2 STEREO CHANNEL INPUT/OUTPUT FRAMEWORK FOR ANDROID PLATFORMS ..................................................................................................................................6 2.1 Motivation ................................................................................................................6 2.2 The Proposed Framework ........................................................................................7 2.2.1 Java Module of the Framework ....................................................................8 2.2.1.1 Graphical User Interface (GUI) .......................................................9 2.2.1.2 Wave Recorder Class .......................................................................9 2.2.1.3 Processing Class.............................................................................11 2.2.1.4 Java Native Interface (JNI) ............................................................11 2.2.2 C/C++ Module of the Framework ..............................................................12 2.2.2.1 Audio Main ....................................................................................13 2.2.2.2 Signal Processing C/C++ Class .....................................................13 2.3 Experiment and Results .........................................................................................14 2.4 Summary ................................................................................................................17 CHAPTER 3 TWO MICROPHONE BASED SPEECH SOURCE LOCALIZATION ..............18 3.1 Motivation ..............................................................................................................18 vii 3.2 Literature Review...................................................................................................19 3.2.1 Direction of Arrival Estimation using GCC ...............................................19 3.2.2 Direction of Arrival Estimation using Frequency Weighted GCC ............21 3.2.3 Direction of Arrival (DOA) Estimation on Smartphone/Tablet ................23 3.3 Proposed DOA Estimation Method .......................................................................25 3.3.1 Pre-Processing Module ..............................................................................25 3.3.2 DOA Estimation Module ...........................................................................28 3.3.3 Post-Processing Module .............................................................................29 3.3.4 Proposed DOA Estimation Method and Sound Source-Tracking ..............30 3.4 Experimental Evaluations ..........................................................................30 3.4.1 Noisy Dataset and Experimental Setup ......................................................30 3.4.2 Evaluation Metrics .....................................................................................33 3.4.3 Results and Discussions .............................................................................34 3.5 Summary ................................................................................................................45 CHAPTER 4 REAL TIME IMPLEMENTATION OF DOA ESTIMATION METHOD ON ANDROID PLATFORMS ............................................................................................................47 4.1 Motivation ..............................................................................................................47 4.2 Shortcomings of Real Time Implementation of SSL on Mobile Platforms...........47 4.3 Real-Time Implementation of Proposed SSL Algorithm ......................................48 4.3.1 DOA Estimation Application Version 1 ....................................................48 4.3.2 DOA Estimation Application Version 2 ....................................................50 4.3.3 DOA Estimation Application Version 3 ....................................................52 4.3.4 DOA Estimation Application Version 4 ....................................................54 4.4 Summary ................................................................................................................59 CHAPTER 5 CONCLUSION ......................................................................................................60 REFERENCES ..............................................................................................................................62