Journal of Computer Science 9 (11): 1496-1505, 2013 ISSN: 1549-3636 © 2013 Science Publications doi:10.3844/jcssp.2013.1496.1505 Published Online 9 (11) 2013 (http://www.thescipub.com/jcs.toc) A REVIEW ON THE DEVELOPMENT OF INDONESIAN SIGN LANGUAGE RECOGNITION SYSTEM 1,2 Sutarman, 1Mazlina Abdul Majid and 1Jasni Mohamad Zain 1Department of Software Engineering, Universiti Malaysia Pahang, Kuantan, Pahang, Malaysia 2Department of Information Technology and Business, University Technology of Yogyakarta, Indonesia Received 2013-06-02, Revised 2013-07-23; Accepted 2013-09-27 ABSTRACT Sign language is mainly employed by hearing-impaired people to communicate with each other. However, communication with normal people is a major handicap for them since normal people do not understand their sign language. Sign language recognition is needed for realizing a human oriented interactive system that can perform an interaction like normal communication. Sign language recognition basically uses two approaches: (1) computer vision-based gesture recognition, in which a camera is used as input and videos are captured in the form of video files stored before being processed using image processing; (2) approach based on sensor data, which is done by using a series of sensors that are integrated with gloves to get the motion features finger grooves and hand movements. Different of sign languages exist around the world, each with its own vocabulary and gestures. Some examples are American Sign Language (ASL), Chinese Sign Language (CSL), British Sign Language (BSL), Indonesian Sign Language (ISL) and so on. The structure of Indonesian Sign Language (ISL) is different from the sign language of other countries, in that words can be formed from the prefix and or suffix. In order to improve recognition accuracy, researchers use methods, such as the hidden Markov model, artificial neural networks and dynamic time warping. Effective algorithms for segmentation, matching the classification and pattern recognition have evolved. The main objective of this study is to review the sign language recognition methods in order to choose the best method for developing the Indonesian sign language recognition system. Keywords: Indonesian Sign Language, Recognition System, Hidden Markov Model, Artificial Neural Network, Dynamic Time Warping 1. INTRODUCTION with sign language will have difficulty to communicate. Thus, software that transcribes symbols Sign language is one of the most important and in sign languages into plain text can help with real-time natural communication modalities. It is a static communication and may also provide interactive expression system that is composed of signs by using training for people to learn a sign language. hand motion aided by facial expressions. Sign language Gesture recognition has become an important is mainly employed by hearing-impaired people to research field with the current focus on interactive communicate with each other. However, communication emotion recognition and hand gesture recognition. with normal people is a major handicap for them since Kinect, as an XBOX motion sense game controller, can normal people do not understand their sign language. be used to obtain to garner video with depth information Sign language recognition is needed for realizing as well as track the skelatal movements of the gamer. a human oriented interactive system, which can Now the Kinect can also be used to recognize body perform an interaction like normal communication. motion without connecting to an Xbox because it can be Without a translator, most people who are not familiar connected to a normal PC to collect information. Corresponding Author: Sutarman, Department of Software Engineering, Universiti Malaysia Pahang, Kuantan, Pahang, Malaysia Science Publications 1496 JCS Sutarman et al. / Journal of Computer Science 9 (11): 1496-1505, 2013 Research in the field of sign language can be produce it, it is the common type of language among categorized into two; one that employs a vision-based deaf people (Lang et al ., 2011). computer (computer vision), while the other is based on the sensor data (Ma et al ., 2000; Maraqa and Abu-Zaiter, 2.2. Indonesian Sign Language (ISL) 2008). In computer vision-based gesture recognition, a The Indonesian Sign Language (ISL), which is also camera is used as input. Videos are captured in the form known as Sistem Isyarat Bahasa Indonesia (SIBI), can be of video files stored before being processed using image broadly divided into two, namely, the alphabet gesture processing (Brashear et al ., 2003; Hninn and Maung, and the word gesture. For the alphabet gesture, SIBI 2009; Stergiopoulou and Papamarkos, 2009). refers to the American Sign Language (ASL), while for Several works on sign language system have been the word gesture the word in Indonesian is used to previously proposed, which mostly make use of symbolize its common meaning. The alphabet gesture is probabilistic models, such as Hidden Markov Models usually used to spell names or words that are not listed in and Artificial Neural Networks. Thus, the main the dictionary of the vocabulary. Word gestures are more objective of this study is to review and compare the widely used in practice and have a much larger sign. performance of sign language recognition methods. Both gesture signs have components to the gesture. The The purpose of the comparison is to find out the best main component to the signal is formed by the fingers sign recognition method for developing the Indonesian and hand movements. The word gesture frequently uses Sign Language (ISL). a hand gesture, instead of a finger formation. Types of gesture in ISL include: (a) basic gesture 2. A REVIEW OF THE SIGN LANGUAGE (the basic word) as depicted as Fig. 1 , (b) additional SYSTEM gesture (prefix, suffix) as depicted in Fig. 2 , (c) formation of the gesture (combined gesture) as depicted 2.1. Sign Language in Fig. 3 , (d) finger alphabet (Character/Numeric) as The term sign language is similar to the language depicted in Fig. 4 . term, in that there are many of both spread throughout Dictionary ISL (DPN, 2001) has been accepted by the various territories of the world. Just like other the Ministry of Education of Indonesia to be used as languages, sign language was developed a long time ago the national standard. ISL standardized it as one of the and has developed over a long time, has a sign language media to help communication among the deaf within grammar and vocabulary and, hence, is considered a real the larger society. ISL consists of systematic rules for language (Braem, 1995). gestures of fingers, hands and other movements that This language is commonly used in deaf symbolize the Indonesian vocabulary. This standard communities, including by interpreters, friends and was accepted based on some considerations, such as families of the deaf, as well as people who are hard of easiness, properness and accuracy of the meanings and hearing themselves. However, these languages are not structure of the language. commonly known outside of these communities and The research on Indonesian sign language is still therefore, communication barriers exist between deaf very little and still need development. The research is and hearing people. still largely in the form of applications, while for studies Sign language communication is multimodal. It using recognition methods such as HMM, ANN, DTW is involves both hand gestures (i.e., manual signing) as well still little and results accuracy needs to be improved. as non-manual signals. Gestures in sign language are defined as specific patterns or movements of the hands, face or body to make our expressions. Since it is a natural language, sign language is closely linked with the culture of the deaf, from which it originates. Thus, knowledge of the culture is necessary to fully understand sign language. The difference between sign language and common language concerns the method to communicate/articulate information. Because no sense of hearing is required to understand sign language and no voice is required to Fig. 1. Example of basic gestures Science Publications 1497 JCS Sutarman et al. / Journal of Computer Science 9 (11): 1496-1505, 2013 Research objects with SIBI such as: The Indonesian Sign Language for the Deaf Computer Application developed by Yanuardi et al . (2010) and Asriani and Susilawati (2010) used the Backpropagation Neural Networks (BNN) method for the Indonesian Sign Language (ISL) recognition, Sekar (2001) used the Neural Networks (NN) method for the Indonesian Sign Language (ISL), Iqbal et al . (2011) used DTW for the Indonesian Sign Language based- sensor accelerometer and sensor flex. Fig. 2. Examples additional gestures (prefix, suffix) 3. SIGN CAPTURING METHODS 3.1. Data Glove Approach One of the traditional sign capturing method is data glove approach ( Fig. 5 ). These methods employ mechanical or optical sensors attached to a glove that transforms finger flexions into electrical signals to determine the hand posture (Xingyan, 2003). This method requires the glove must be worn and a Fig. 3. Examples of combined gesture (berlompatan) wearisome device with a load of cables connected to the computer, which will hamper the naturalness of user- computer interaction. However, the disadvantage of this method is it requires the glove to be worn, which is a wearisome device with many cables connected into the computer that will hamper the naturalness of the user computer interaction (Mitra and Acharya, 2007). 3.2. Vision-based Sign Extraction Another common sign capturing method is vision- based sign extraction. This method is usually done by capturing an input image using a camera ( Fig. 6 ). In order to create a database for a gesture system, the gestures should be selected with their relevant meaning, in which each gesture may contain multi samples (Hasan and Mishra, 2010) to increase the accuracy of the system. Vision-based method is widely deployed for sign language recognition. Sign gestures are captured by a fixed camera in front of signers.