DOCSLIB.ORG

Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph Jun Wang Ashok Samal Jordan R. Green Dept. of Bioengineering Dept. of Computer Science & Dept. of Communication Sci- Callier Center for Communi- Engineering ences & Disorders cation Disorders University of Nebraska- MGH Institute of Health Pro- University of Texas at Dallas Lincoln fessions [email protected] [email protected] [email protected] treatment options for these individuals including Abstract (1) esophageal speech, which involves oscillation of the esophagus and is difficult to learn; (2) tra- A silent speech interface (SSI) maps articula- cheo-esophageal speech, in which a voice pros- tory movement data to speech output. Alt- thesis is placed in a tracheo-esophageal puncture; hough still in experimental stages, silent and (3) electrolarynx, an external device held on speech interfaces hold significant potential the neck during articulation, which produces a for facilitating oral communication in persons robotic voice quality (Liu and Ng, 2007). Per- after laryngectomy or with other severe voice impairments. Despite the recent efforts on si- haps the greatest disadvantage of these ap- lent speech recognition algorithm develop- proaches is that they produce abnormal sounding ment using offline data analysis, online test speech with a fundamental frequency that is low of SSIs have rarely been conducted. In this and limited in range. The abnormal voice quality paper, we present a preliminary, online test of output severely affects the social life of people a real-time, interactive SSI based on electro- after laryngectomy (Liu and Ng, 2007). In addi- magnetic motion tracking. The SSI played tion, the tracheo-esophageal option requires an back synthesized speech sounds in response additional surgery, which is not suitable for eve- to the user’s tongue and lip movements. ry patient (Bailey et al., 2006). Although re- Three English talkers participated in this test, search is being conducted on improving the where they mouthed (silently articulated) phrases using the device to complete a voice quality of esophageal or electrolarynx phrase-reading task. Among the three partici- speech (Doi et al., 2010; Toda et al., 2012), new pants, 96.67% to 100% of the mouthed assistive technologies based on non-audio infor- phrases were correctly recognized and corre- mation (e.g., visual or articulatory information) sponding synthesized sounds were played af- may be a good alternative approach for providing ter a short delay. Furthermore, one participant natural sounding speech output for persons after demonstrated the feasibility of using the SSI laryngectomy. for a short conversation. The experimental re- Visual speech recognition (or automatic lip sults demonstrated the feasibility and poten- reading) typically uses an optical camera to ob- tial of silent speech interfaces based on elec- tain lip and/or facial features during speech (in- tromagnetic articulograph for future clinical applications. cluding lip contour, color, opening, movement, etc.) and then classify these features to speech units (Meier et al., 2000; Oviatt, 2003). Howev- 1 Introduction er, due to the lack of information from tongue, Daily communication is often a struggle for per- the primary articulator, visual speech recognition sons who have undergone a laryngectomy, a sur- (i.e., using visual information only, without gical removal of the larynx due to the treatment tongue and audio information) may obtain a low of cancer (Bailey et al., 2006). In 2013, about accuracy (e.g., 30% - 40% for phoneme classifi- 12,260 new cases of laryngeal cancer were esti- cation, Livescu et al., 2007). Furthermore, Wang mated in the United States (American Cancer and colleagues (2013b) have showed any single Society, 2013). Currently, there are only limited tongue sensor (from tongue tip to tongue body 38 Proceedings of the 5th Workshop on Speech and Language Processing for Assistive Technologies (SLPAT), pages 38–45, Baltimore, Maryland USA, August 26 2014. c 2014 Association for Computational Linguistics Figure 1. Design of the real-time silent speech interface. back on the midsagittal line) encodes significant- based SSIs have been tested online with multiple ly more information in distinguishing phonemes subjects and encouraging results were obtained than do lips. However, visual speech recognition in a phrase reading task where the subjects were is well suited for applications with small- asked to silently articulate sixty phrases (Denby vocabulary (e.g., a lip-reading based command- et al., 2011). SSI based on electromagnetic sens- and-control system for home appliance) or using ing has been only tested using offline analysis visual information as an additional source for (using pre-recorded data) collected from single acoustic speech recognition, referred to as audio- subjects (Fagan et al., 2008; Hofe et al., 2013), visual speech recognition (Potamianos et al., although some work simulated online testing 2003), because such a system based on portable using prerecorded data (Wang et al., 2012a, camera is convenient in practical use. In contrast, 2012b, 2013c). Online tests of SSIs using elec- SSIs, with tongue information, have potential to tromagnetic articulograph with multiple subjects obtain a high level of silent speech recognition are needed to show the feasibility and potential accuracy (without audio information). Currently, of the SSIs for future clinical applications. two major obstacles for SSI development are In this paper, we report a preliminary, online lack of (a) fast and accurate recognition algo- test of a newly-developed, real-time, and interac- rithms and (b) portable tongue motion tracking tive SSI based on a commercial EMA. EMA devices for daily use. tracks articulatory motion by placing small sen- SSIs convert articulatory information into text sors on the surface of tongue and other articula- that drives a text-to-speech synthesizer. Although tors (e.g., lips and jaw). EMA is well suited for still in developmental stages (e.g., speaker- the early stage of SSI development because it (1) dependent recognition, small-vocabulary), SSIs is non-invasive, (2) has a high spatial resolution even have potential to provide speech output in motion tracking, (3) has a high sampling rate, based on prerecorded samples of the patient’s and (4) is affordable. In this experiment, partici- own voice (Denby et al., 2010; Green et al., pants used the real-time SSI to complete an 2011; Wang et al., 2009). Potential articulatory online phrase-reading task and one of them had a data acquisition methods for SSIs include ultra- short conversation with another person. The re- sound (Denby et al., 2011; Hueber et al., 2010), sults demonstrated the feasibility and potential of surface electromyography electrodes (Heaton et SSIs based on electromagnetic sensing for future al., 2011; Jorgensen and Dusan, 2010), and elec- clinical applications. tromagnetic articulograph (EMA) (Fagan et al., 2008; Wang et al., 2009, 2012a). 2 Design Despite the recent effort on silent speech in- 2.1 Major design terface research, online test of SSIs has rarely been studied. So far, most of the published work Figure 1 illustrates the three-component design on SSIs has focused on development of silent of the SSI: (a) real-time articulatory motion speech recognition algorithm through offline tracking using a commercial EMA, (b) online analysis (i.e., using prerecorded data) (Fagan et silent speech recognition (converting articulation al., 2008; Heaton et al., 2011; Hofe et al., 2013; information to text), and (c) text-to-speech syn- Hueber et al., 2010; Jorgenson et al., 2010; Wang thesis for speech output. et al., 2009a, 2012a, 2012b, 2013c). Ultrasound- The EMA system (Wave Speech Research 39 Figure 2. Demo of a participant using the silent speech interface. The left picture illustrates the coordinate system and sensor locations (sensor labels are described in text); in the right picture, a participant is using the silent speech interface to finish the online test. system, Northern Digital Inc., Waterloo, Canada) phrases into the system through the GUI. Adding was used to track the tongue and lip movement a new phrase in the vocabulary is done in two in real-time. The sampling rate of the Wave sys- steps. The user (the patient) first enters the tem was 100 Hz, which is adequate for this ap- phrase using a keyboard (keyboard input can also plication (Wang et al., 2012a, 2012b, 2013c). be done by an assistant or speech pathologist), The spatial accuracy of motion tracking using and then produces a few training samples for the Wave is 0.5 mm (Berry, 2011). phrase (a training sample is articulatory data la- The online recognition component recognized beled with a phrase). The system automatically functional phrases from articulatory movements re-trains the recognition model integrating the in real-time. The recognition component is mod- newly-added training samples. Users can delete ular such that alternative classifiers can easily invalid training samples using the GUI as well. replace and be integrated into the SSI. In this preliminary test, recognition was speaker- 2.3 Real-time data processing dependent, where training and testing data were The tongue and lip movement positional data from the same speakers. obtained from the Wave system were processed The third component played back either pre- in real-time prior to being used for recognition. recorded or synthesized sounds using a text-to- This included the calculation of head- speech synthesizer (Huang et al., 1997). independent positions of the tongue and lip sen- sors and low pass filtering for removing noise. 2.2 Other designs The movements of the 6 DOF head sensor A graphical user interface (GUI) is integrated were used to calculate the head-independent into the silent speech interface for ease of opera- movements of other sensors. The Wave system tion. Using the GUI, users can instantly re-train represents object orientation or rotation (denoted the recognition engine (classifier) when new by yaw, pitch, and roll in Euler angles) in qua- training samples are available.
Load more

Recommended publications
  • Silent Speech Interfaces B
    Silent Speech Interfaces B. Denby, T. Schultz, K. Honda, Thomas Hueber, J.M. Gilbert, J.S. Brumberg To cite this version: B. Denby, T. Schultz, K. Honda, Thomas Hueber, J.M. Gilbert, et al.. Silent Speech Interfaces. Speech Communication, Elsevier : North-Holland, 2010, 52 (4), pp.270. 10.1016/j.specom.2009.08.002. hal- 00616227 HAL Id: hal-00616227 https://hal.archives-ouvertes.fr/hal-00616227 Submitted on 20 Aug 2011 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Accepted Manuscript Silent Speech Interfaces B. Denby, T. Schultz, K. Honda, T. Hueber, J.M. Gilbert, J.S. Brumberg PII: S0167-6393(09)00130-7 DOI: 10.1016/j.specom.2009.08.002 Reference: SPECOM 1827 To appear in: Speech Communication Received Date: 12 April 2009 Accepted Date: 20 August 2009 Please cite this article as: Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J.M., Brumberg, J.S., Silent Speech Interfaces, Speech Communication (2009), doi: 10.1016/j.specom.2009.08.002 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript.
    [Show full text]
  • Text-To-Speech Synthesis As an Alternative Communication Means
    Biomed Pap Med Fac Univ Palacky Olomouc Czech Repub. 2021 Jun; 165(2):192-197. Text-to-speech synthesis as an alternative communication means after total laryngectomy Barbora Repovaa#, Michal Zabrodskya#, Jan Plzaka, David Kalferta, Jindrich Matousekb, Jan Betkaa Aims. Total laryngectomy still plays an essential part in the treatment of laryngeal cancer and loss of voice is the most feared consequence of the surgery. Commonly used rehabilitation methods include esophageal voice, electrolarynx, and implantation of voice prosthesis. In this paper we focus on a new perspective of vocal rehabilitation utilizing al- ternative and augmentative communication (AAC) methods. Methods and Patients. 61 consecutive patients treated by means of total laryngectomy with or w/o voice prosthesis implantation were included in the study. All were offered voice banking and personalized speech synthesis (PSS). They had to voluntarily express their willingness to participate and to prove the ability to use modern electronic com- munication devices. Results. Of 30 patients fulfilling the study criteria, only 18 completed voice recording sufficient for voice reconstruction and synthesis. Eventually, only 7 patients started to use this AAC technology during the early postoperative period. The frequency and total usage time of the device gradually decreased. Currently, only 6 patients are active users of the technology. Conclusion. The influence of communication with the surrounding world on the quality of life of patients after total laryngectomy is unquestionable. The possibility of using the spoken word with the patient's personalized voice is an indisputable advantage. Such a form of voice rehabilitation should be offered to all patients who are deemed eligible.
    [Show full text]
  • Across-Speaker Articulatory Normalization for Speaker-Independent Silent Speech Recognition Jun Wang University of Texas at Dallas, [email protected]
    University of Nebraska - Lincoln DigitalCommons@University of Nebraska - Lincoln CSE Conference and Workshop Papers Computer Science and Engineering, Department of Fall 9-2014 Across-speaker Articulatory Normalization for Speaker-independent Silent Speech Recognition Jun Wang University of Texas at Dallas, [email protected] Ashok Samal University of Nebraska-Lincoln, [email protected] Jordan Green MGH Institute of Health Professions, [email protected] Follow this and additional works at: http://digitalcommons.unl.edu/cseconfwork Part of the Computer Sciences Commons, and the Speech and Hearing Science Commons Wang, Jun; Samal, Ashok; and Green, Jordan, "Across-speaker Articulatory Normalization for Speaker-independent Silent Speech Recognition" (2014). CSE Conference and Workshop Papers. Paper 234. http://digitalcommons.unl.edu/cseconfwork/234 This Article is brought to you for free and open access by the Computer Science and Engineering, Department of at DigitalCommons@University of Nebraska - Lincoln. It has been accepted for inclusion in CSE Conference and Workshop Papers by an authorized administrator of DigitalCommons@University of Nebraska - Lincoln. INTERSPEECH 2014 Across-speaker Articulatory Normalization for Speaker-independent Silent Speech Recognition Jun Wang 1, 2, Ashok Samal 3, Jordan R. Green 4 1 Department of Bioengineering 2 Callier Center for Communication Disorders University of Texas at Dallas, Dallas, Texas, USA 3 Department of Computer Science & Engineering University of Nebraska-Lincoln, Lincoln, Nebraska, USA
    [Show full text]
  • Sessions, Workshops, and Technical Initiatives
    TUESDAY MORNING, 19 MAY 2015 BALLROOM 3, 8:00 A.M. TO 12:00 NOON Session 2aAAa Architectural Acoustics, Noise, Speech Communication, Psychological Physiological Acoustics, and Signal Processing in Acoustics: Sixty-Fifth Anniversary of Noise and Health: Session in Honor of Karl Kryter I David M. Sykes, Cochair Architectural Acoustics, Rensselaer Polytechnic Institute, 31 Baker Farm Rd., Lincoln, MA 01773 Michael J. Epstein, Cochair Northeastern University, 360 Huntington Ave., 226FR, Boston, MA 02115 William J. Cavanaugh, Cochair Cavanaugh Tocci Assoc. Inc., 3 Merifield Ln., Natick, MA 01760-5520 Chair’s Introduction—8:00 Invited Papers 8:05 2aAAa1. Auditory and non-auditory effects of noise on health: An ICBEN perspective. Mathias Basner (Psychiatry, Univ. of Penn- sylvania Perelman School of Medicine, 1019 Blockley Hall, 423 Guardian Dr., Philadelphia, PA 19104-6021, [email protected]) Noise is pervasive in everyday life and induces both auditory and non-auditory health effects. Noise-induced hearing loss remains the most common occupational disease in the United States, but is also increasingly caused by social noise exposure (e.g., through music players). Simultaneously, evidence on the public health impact of non-auditory effects of environmental noise exposure is growing. According to the World Health Organization (WHO), ca. 1.6 Million healthy life years (DALYs) are lost annually in the western mem- ber states of the European Union due to exposure to environmental noise. The majority (>90%) of these effects can be attributed to noise-induced sleep disturbance and community annoyance, but noise may also interfere with communication and lead to cognitive impairment of children. Epidemiological studies increasingly support an association of long-term noise exposure with the incidence of hypertension and cardiovascular disease.
    [Show full text]
  • Direct Generation of Speech from Electromyographic Signals: EMG-To
    EMG-to-Speech: Direct Generation of Speech from Facial Electromyographic Signals Zur Erlangung des akademischen Grades eines Doktors der Ingenieurwissenschaften der KIT-Fakult¨atf¨urInformatik des Karlsruher Instituts f¨urTechnologie (KIT) genehmigte Dissertation von Matthias Janke aus Offenburg Tag der m¨undlichen Pr¨ufung: 22. 04. 2016 Erste Gutachterin: Prof. Dr.-Ing. Tanja Schultz Zweiter Gutachter: Prof. Alan W Black, PhD Summary For several years, alternative speech communication techniques have been examined, which are based solely on the articulatory muscle signals instead of the acoustic speech signal. Since these approaches also work with completely silent articulated speech, several advantages arise: the signal is not corrupted by background noise, bystanders are not disturbed, as well as assistance to people who have lost their voice, e.g. due to accident or due to disease of the larynx. The general objective of this work is the design, implementation, improve- ment and evaluation of a system that uses surface electromyographic (EMG) signals and directly synthesizes an audible speech output: EMG-to-speech. The electrical potentials of the articulatory muscles are recorded by small electrodes on the surface of the face and neck. An analysis of these signals allows interpretations on the movements of the articulatory apparatus and in turn on the spoken speech itself. An approach for creating an acoustic signal from the EMG-signal is the usage of techniques from automatic speech recognition. Here, a textual output is produced, which in turn is further processed by a text-to-speech synthesis component. However, this approach is difficult resulting from challenges in the speech recognition part, such as the restriction to a given vocabulary or recognition errors of the system.
    [Show full text]
  • A Comparison of Esophageal and Electrolarynx Speech Using Audio
    A Comparison of Esophageal and Electrolarynx Speech Using Audio and Audiovisual Recordings An Honors Thesis (ID 499) Mary L. Salzmann Dr. William Kramer -, Ball State University Muncie, Indiana May 1987 Spring Quarter 1987 - S')I.· .... C' ,II ,.'1re/, .', , I. ¥- TABLE OF CONTENTS Page I. Introduction • 1 II. Review of Related Literature • 2 Esophageal speech • 2 Electrolarynx speech 3 Research 5 Quality of alaryngeal speech 9 III. Method • • 12 Subjects • 12 Instrumentation • • 12 Procedures . 12 Statistical design and analysis of data •• 13 IV. Results · 14 Charts · 16 V. Discussion • · 17 VI. References • 18 INTRODUCTION There are several a1aryngea1 speech methods presently available to the laryngectomized popu1ation--esophagea1 speech, the e1ectro1arynx, and the tracheoesophageal puncture, for example. These methods are used with varying degrees of satisfaction by the patients. To the average college-age student who has had no previous contact with 1aryngectomees, however, the speech methods may be met with a wide range of reactions and preferences. The purpose of this study is to survey the listener satisfaction of a group of college-age students who have had no previous contact with a1aryngea1 speech to determine their reactions and ratings of esophageal speech as compared to electro larynx speech. Comparisons will also be made between the data obtained from audiovisual presentation and from audio presentation only. Several studies have been performed to collect data comparing the acceptability of specific a1aryngea1 speech methods. Of these, many of the subjects chosen to rate the a1aryngea1 speaker were speech-language pathologists. Clinicians in the field of speech-language pathology may already have strong preferences for a particular a1aryngea1 speech method which will influence their ratings.
    [Show full text]
  • Biosignal Sensors and Deep Learning-Based Speech Recognition: a Review
    sensors Review Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review Wookey Lee 1 , Jessica Jiwon Seong 2, Busra Ozlu 3, Bong Sup Shim 3, Azizbek Marakhimov 4 and Suan Lee 5,* 1 Biomedical Science and Engineering & Dept. of Industrial Security Governance & IE, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 2 Department of Industrial Security Governance, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 3 Biomedical Science and Engineering & Department of Chemical Engineering, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] (B.O.); [email protected] (B.S.S.) 4 Frontier College, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 5 School of Computer Science, Semyung University, Jecheon 27136, Korea * Correspondence: [email protected] Abstract: Voice is one of the essential mechanisms for communicating and expressing one’s intentions as a human being. There are several causes of voice inability, including disease, accident, vocal abuse, medical surgery, ageing, and environmental pollution, and the risk of voice loss continues to increase. Novel approaches should have been developed for speech recognition and production because that would seriously undermine the quality of life and sometimes leads to isolation from society. In this review, we survey mouth interface technologies which are mouth-mounted devices for speech recognition, production, and volitional control, and the corresponding research to de- velop artificial mouth technologies based on various sensors, including electromyography (EMG), electroencephalography (EEG), electropalatography (EPG), electromagnetic articulography (EMA), permanent magnet articulography (PMA), gyros, images and 3-axial magnetic sensors, especially Citation: Lee, W.; Seong, J.J.; Ozlu, B.; with deep learning techniques.
    [Show full text]
  • Silent Speech Interfaces for Speech Restoration: a Review Jose A
    1 Silent Speech Interfaces for Speech Restoration: A Review Jose A. Gonzalez-Lopez, Alejandro Gomez-Alanis, Juan M. Mart´ın-Donas,˜ Jose´ L. Perez-C´ ordoba,´ and Angel M. Gomez Abstract—This review summarises the status of silent speech survey conducted in 70 countries, concluded that 3.6% of the interface (SSI) research. SSIs rely on non-acoustic biosignals population had severe to extreme difficulty with participation generated by the human body during speech production to in the community, a condition which includes communication enable communication whenever normal verbal communication is not possible or not desirable. In this review, we focus on the impairment as a specific case. first case and present latest SSI research aimed at providing Speech and language impairments have a profound impact new alternative and augmentative communication methods for on the lives of people who suffer them, leading them to persons with severe speech disorders. SSIs can employ a variety struggle with daily communication routines. Besides, many of biosignals to enable silent communication, such as electro- service and health-care providers are not trained to inter- physiological recordings of neural activity, electromyographic (EMG) recordings of vocal tract movements or the direct tracking act with speech-disabled persons, and feel uncomfortable or of articulator movements using imaging techniques. Depending ineffective in communicating with them, which aggravates on the disorder, some sensing techniques may be better suited the stigmatisation of this population [4]. As a result, people than others to capture speech-related information. For instance, with speech impairments often develop feelings of personal EMG and imaging techniques are well suited for laryngectomised isolation and social withdrawal, which can lead to clinical patients, whose vocal tract remains almost intact but are unable to speak after the removal of the vocal folds, but fail for depression [5]–[11].
    [Show full text]