Gesture Based Interface for Asynchronous Video Communication for Deaf People in South Africa By Tshifhiwa Ramuhaheli Supervised by Prof. Edwin Blake Thesis presented for the degree of Master of Science In the Department of Computer Science University of Cape Town May 2010 Copyright © 2010 By Tshifhiwa Ramuhaheli i Abstract The preferred method of communication amongst Deaf people is that of sign language. There are problems with the video quality when using the real-time video communication available on mobile phones. The alternative is to use text-based communication on mobile phones, however findings from other research studies show that Deaf people prefer using sign language to communicate with each other rather than text. This dissertation looks at implementing a gesture-based interface for an asynchronous video communication for Deaf people. The gesture interface was implemented on a store and forward video architecture since this preserves the video quality even when there is low bandwidth. In this dissertation three gesture-based video communication prototypes were designed and implemented using a user centred design approach. These prototypes were implemented on both the computer and mobile devices. The first prototype was computer based and the evaluation of this prototype showed that the gesture based interface improved the usability of sign language video communication. The second prototype is set up on the mobile device and it was tested on several mobile devices but the device limitation made it impossible to support all the features needed in the video communication. The different problems experienced on the dissimilar devices made the task of implementing the prototypes on the mobile platform challenging. The prototype was revised several times before it was tested on a different mobile phone. The final prototype used both the mobile phone and the computer. The computer served to simulate a mobile device with greater processing power. This approach simulated a more powerful future mobile device capable of running the gesture-based interface. The computer was used for video processing but to the user it was as if the whole system was running on the mobile phone. The evaluation process was conducted with ten Deaf users in order to determine the efficiency and usability of the prototype. The results showed that the majority of the users were satisfied with the quality of the video communication. The evaluation also revealed usability problems but the benefits of communicating in sign language outweighed the usability difficulties. Furthermore the users were more interested in the video communication on the mobile devices than on the computer as this was a much more familiar technology and offered the convenience of mobility. ii Table of Contents Abstract ..................................................................................................................................... ii Acknowledgements .................................................................................................................. x 1. Introduction ...................................................................................................................... 1 1.1 Sign Language ............................................................................................................. 2 1.2 Motivation ................................................................................................................... 3 1.3 Research Objectives .................................................................................................... 3 1.4 Significance of Research ............................................................................................. 4 1.4.1 Research Questions .............................................................................................. 4 1.5 Dissertation Outline..................................................................................................... 6 2. Background Chapter ....................................................................................................... 7 2.1 Deaf ............................................................................................................................. 7 2.2 Deaf Communication .................................................................................................. 8 2.3 Video Communication ................................................................................................ 8 2.3.1 Video on Mobile Devices .................................................................................... 9 2.3.2 Video Compression .............................................................................................. 9 2.4 Gesture Recognition .................................................................................................. 10 2.4.1 Sign Language Recognition ............................................................................... 12 2.5 Mobile Device ........................................................................................................... 13 2.5.1 Mobile Devices in ICT4D .................................................................................. 13 2.5.2 Mobile Device Limitations ................................................................................ 14 2.5.3 Development Environment ................................................................................ 15 2.5.4 Image Processing Performance on Mobile Phones ............................................ 19 2.6 Chapter Summary ...................................................................................................... 21 3. Experimental Design ...................................................................................................... 22 3.1 Research Questions ................................................................................................... 22 iii 3.2 Methodology ............................................................................................................. 22 3.2.1 Requirements Gathering .................................................................................... 23 3.2.2 Computer Prototype ........................................................................................... 25 3.2.3 First Mobile Phone Prototype ............................................................................ 25 3.2.4 Second Mobile Phone Prototype ........................................................................ 26 3.3 Prototype Designs ..................................................................................................... 26 3.4 Evaluation of Designs ............................................................................................... 27 3.5 Participants ................................................................................................................ 28 3.6 Technology and Tools ............................................................................................... 29 3.7 Chapter Summary ...................................................................................................... 30 4. Understanding Users ..................................................................................................... 31 4.1 Requirements Gathering ............................................................................................ 31 4.1.1 Focus Group Study ............................................................................................ 31 4.1.2 User Feedback .................................................................................................... 34 4.2 Communication Survey ............................................................................................. 36 4.2.1 Mobile Phone Ownership .................................................................................. 37 4.2.2 Mobile Phone Usage .......................................................................................... 38 4.3 Chapter Summary ...................................................................................................... 39 5. Design and Implementation .......................................................................................... 40 5.1 Design........................................................................................................................ 40 5.1.1 System Requirements......................................................................................... 41 5.1.2 Interface Design ................................................................................................. 44 5.2 Computer Prototype Design ...................................................................................... 45 5.2.1 Computer Prototype Evaluation ......................................................................... 48 5.3 Mobile Phone Design ................................................................................................ 53 5.3.1 Changes Made .................................................................................................... 54 5.3.2 Mobile Phone Design Problems......................................................................... 56 iv 5.4 Mobile Phone Simulation Design ............................................................................. 61 5.4.1 The Mobile Application ..................................................................................... 62 5.4.2 The Computer Application ................................................................................ 62 5.4.3 Task Flow..........................................................................................................