DOCSLIB.ORG

Biosignal Sensors and Deep Learning-Based Speech Recognition: a Review

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text

Load more

sensors Review Biosignal Sensors and Deep Learning-Based Speech Recognition: A Review Wookey Lee 1 , Jessica Jiwon Seong 2, Busra Ozlu 3, Bong Sup Shim 3, Azizbek Marakhimov 4 and Suan Lee 5,* 1 Biomedical Science and Engineering & Dept. of Industrial Security Governance & IE, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 2 Department of Industrial Security Governance, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 3 Biomedical Science and Engineering & Department of Chemical Engineering, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] (B.O.); [email protected] (B.S.S.) 4 Frontier College, Inha University, 100 Inharo, Incheon 22212, Korea; [email protected] 5 School of Computer Science, Semyung University, Jecheon 27136, Korea * Correspondence: [email protected] Abstract: Voice is one of the essential mechanisms for communicating and expressing one’s intentions as a human being. There are several causes of voice inability, including disease, accident, vocal abuse, medical surgery, ageing, and environmental pollution, and the risk of voice loss continues to increase. Novel approaches should have been developed for speech recognition and production because that would seriously undermine the quality of life and sometimes leads to isolation from society. In this review, we survey mouth interface technologies which are mouth-mounted devices for speech recognition, production, and volitional control, and the corresponding research to de- velop artificial mouth technologies based on various sensors, including electromyography (EMG), electroencephalography (EEG), electropalatography (EPG), electromagnetic articulography (EMA), permanent magnet articulography (PMA), gyros, images and 3-axial magnetic sensors, especially Citation: Lee, W.; Seong, J.J.; Ozlu, B.; with deep learning techniques. We especially research various deep learning technologies related to Shim, B.S.; Marakhimov, A.; Lee, S. voice recognition, including visual speech recognition, silent speech interface, and analyze its flow, Biosignal Sensors and Deep and systematize them into a taxonomy. Finally, we discuss methods to solve the communication Learning-Based Speech Recognition: A Review. Sensors 2021, 21, 1399. problems of people with disabilities in speaking and future research with respect to deep learning https://doi.org/10.3390/s21041399 components. Academic Editors: Wai Lok Woo and Keywords: mouth interface; voice production; artificial larynx; EMG; biosignal; deep learning Biswanath Samanta Received: 28 December 2020 Accepted: 12 February 2021 Published: 17 February 2021 1. Introduction Voice is a basic means of communication and social interaction through spoken lan- Publisher’s Note: MDPI stays neutral guage. People with voice disorders face serious problems in their daily lives, which may with regard to jurisdictional claims in lead to emotional instability and isolation from society. A voice disorder implies that the published maps and institutional affil- pitch, intensity, or fluidity of one’s voice does not conform to his or her gender, age, body iations. composition, social environment, and geographic location. In other words, the term voice disorder refers to all abnormal conditions in which the expression of the voice is not in its normal range. In the human body, voice is produced by the vibration of the vocal cords as a result of the airflow supplied by the respiratory system, thus the normal voice Copyright: © 2021 by the authors. production depends on the coordination among airflow, laryngeal muscle strength and the Licensee MDPI, Basel, Switzerland. supraglottic resonator cavities such as pharyngeal, oral and nasal cavity [1]. The reasons This article is an open access article for the voice disorder can be categorized as organic, functional, and/or psychogenic causes. distributed under the terms and Organic causes may have structural (vocal fold abnormalities, inflammation or trauma conditions of the Creative Commons to the larynx) and neurologic (recurrent laryngeal nerve paralysis, adductor/abductor Attribution (CC BY) license (https:// spasmodic dysphonia, Parkinson’s disease, multiple sclerosis) origins, while functional creativecommons.org/licenses/by/ causes may arise from phonotrauma, muscle tension dysphonia, ventricular phonation, 4.0/). Sensors 2021, 21, 1399. https://doi.org/10.3390/s21041399 https://www.mdpi.com/journal/sensors Sensors 2021, 21, 1399 2 of 22 and vocal fatigue. Anxiety and depression are considered psychogenic causes of voice disorder [1]. Although there are several approaches for the treatment of voice disorder, such as PMA using articulatory data captured from the lips and tongue [2] or a contact- less silent speech recognition system using an impulse radio ultra-wideband (IR-UWB) radar [3], the communication options are limited for the patients whose larynx (voice box) Sensors 2019, 19, x FOR PEER REVIEWis surgically removed following throat cancer or trauma [4]. To overcome this problem,2 of 20 the external device, called electrolarynx, has been developed as a form of artificial larynx [2]communication or a contactless where silent speech an electronic recognition vibration system is using produced an impulse and resultedradio ultra in-wideband a monotonic (IR- UWB)sound radar that [3], will the be communication formed to speech options [ 4are]. However,limited for the the patients unpractical whose uselarynx and (voice unnatural box) is surgicallyvoice production removed makefollowing the devicethroat cancer challenging or trauma for the[4]. To patients. overcome Thus, this a problem, novel concept the external must device,be developed called electrolarynx, for voice recognition has been developed and production as a form of technologies artificial larynx which communication also can include where an electronic vibration is produced and resulted in a monotonic sound that will be formed to speech brain-computer interfaces (BCIs) and silent-speech interfaces (SSIs). SSI is considered as a [4]. However, the unpractical use and unnatural voice production make the device challenging for theplausible patients. approach Thus, a novel to producing concept must natural-sounding be developed for speech voice recognition by capturing and biosignals production from technol- the ogiesarticulators, which also neural can include pathways, brain- orcomputer the brain interfaces itself in (BCI brain-computers) and silent-speech interfaces interfaces (BCIs) (SSIs [).5 –SSI8]. isRecently, considered various as a plausible biosignals approach captured to producing by techniques natural-sounding such as ultrasound,speech by capturing optical biosignals imagery, fromEPG, the EEG, articulators, and surface neural electromyography pathways, or the brain (sEMG) itself in have brain- beencomputer investigated interfaces in(BCI termss) [5–8]. of Recently,developing various silent biosignals speech captured communication by techniques systems such [ 8as– 10ultrasound,]. optical imagery, EPG, EEG, and surfaceIn this electromyography review, as summarized (sEMG) in have Figure been1 ,investigated we report on in theterms recent of developing advances silent in the speech field communicationof biosignal-based systems voice [8– recognition10]. and production with deep learning-based technologies. We firstIn th introduceis review, sensoras summarized technologies in Figure that 1, can we acquire report on voice-related the recent advances biosignal in data the asfield well of biosignalas their- applicationsbased voice recognition in voice recognitionand production and with production. deep learning Regarding-based technologies. deep learning, We first we introducealso present sensor not technologies only voice recognitionthat can acquire technology voice-related through biosignal sound data or as image well as but their also applica- silent tions in voice recognition and production. Regarding deep learning, we also present not only voice speech representation interface. We concluded with a summary and perspectives on the recognition technology through sound or image but also silent speech representation interface. We current and future applications. concluded with a summary and perspectives on the current and future applications. Figure 1. How to help people with voice disorders communicate with others: to recognize voices, which indicates that biosignals are processed via various methods and and transmitted transmitted to to devices devices suitable suitable for for the the user, user, and can be used in several fields, such as robotics, medical engineering, and image processing. fields, such as robotics, medical engineering, and image processing. 2.2. Biosignal Biosignal-Based-Based Speech Speech Recognition Recognition BiosignalsBiosignals provide provide the theinformation information belong belong to the electrical, to the electrical, chemical, mechanical, chemical,mechanical, and biologi- caland processes biological of the processes living organism of the living[11,12]. organismIn the field [of11 ,biosignal12]. In- thebased field speech of biosignal-based recognition, sig- nalsspeech from recognition, brain, muscle signals and movement from brain, are muscle considered and movementas potentially are useful considered biosignals as potentiallythat can be measured by different techniques. Traditional acoustic sensors capture the sound pressure waves useful biosignals that can be measured by different techniques. Traditional acoustic sensors resulting in acoustic biosignals.
Recommended publications
  • Arxiv:1807.06441V1 [Eess.AS] 12 Jul 2018 Etr Saln Hr-Emmmr LT)Ta a Endesigne Been Has That (LSTM) Memory Popu Short-Term Most Long the a (Rnns)

    Arxiv:1807.06441V1 [Eess.AS] 12 Jul 2018 Etr Saln Hr-Emmmr LT)Ta a Endesigne Been Has That (LSTM) Memory Popu Short-Term Most Long the a (Rnns)

    A Comparison of Adaptation Techniques and Recurrent Neural Network Architectures Jan Vanˇek[0000−0002−2639−6731], Josef Mich´alek[0000−0001−7757−3163], Jan Zelinka[0000−0001−6834−9178], and Josef Psutka[0000−0002−0764−3207] University of West Bohemia Univerzitn´ı8, 301 00 Pilsen, Czech Republic {vanekyj,orcus,zelinka,psutka}@kky.zcu.cz Abstract. Recently, recurrent neural networks have become state-of- the-art in acoustic modeling for automatic speech recognition. The long short-term memory (LSTM) units are the most popular ones. However, alternative units like gated recurrent unit (GRU) and its modifications outperformed LSTM in some publications. In this paper, we compared five neural network (NN) architectures with various adaptation and fea- ture normalization techniques. We have evaluated feature-space maxi- mum likelihood linear regression, five variants of i-vector adaptation and two variants of cepstral mean normalization. The most adaptation and normalization techniques were developed for feed-forward NNs and, ac- cording to results in this paper, not all of them worked also with RNNs. For experiments, we have chosen a well known and available TIMIT phone recognition task. The phone recognition is much more sensitive to the quality of AM than large vocabulary task with a complex language model. Also, we published the open-source scripts to easily replicate the results and to help continue the development. Keywords: neural networks · acoustic model · TIMIT · LSTM · GRU · phone recognition · adaptation · i-vectors 1 Introduction Neural Networks (NNs) and deep NNs (DNNs) became dominant in the field of the acoustic modeling several years ago. Simple feed-forward (FF) DNNs were faded away in recent years.
  • Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph

    Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph

    Preliminary Test of a Real-Time, Interactive Silent Speech Interface Based on Electromagnetic Articulograph Jun Wang Ashok Samal Jordan R. Green Dept. of Bioengineering Dept. of Computer Science & Dept. of Communication Sci- Callier Center for Communi- Engineering ences & Disorders cation Disorders University of Nebraska- MGH Institute of Health Pro- University of Texas at Dallas Lincoln fessions [email protected] [email protected] [email protected] treatment options for these individuals including Abstract (1) esophageal speech, which involves oscillation of the esophagus and is difficult to learn; (2) tra- A silent speech interface (SSI) maps articula- cheo-esophageal speech, in which a voice pros- tory movement data to speech output. Alt- thesis is placed in a tracheo-esophageal puncture; hough still in experimental stages, silent and (3) electrolarynx, an external device held on speech interfaces hold significant potential the neck during articulation, which produces a for facilitating oral communication in persons robotic voice quality (Liu and Ng, 2007). Per- after laryngectomy or with other severe voice impairments. Despite the recent efforts on si- haps the greatest disadvantage of these ap- lent speech recognition algorithm develop- proaches is that they produce abnormal sounding ment using offline data analysis, online test speech with a fundamental frequency that is low of SSIs have rarely been conducted. In this and limited in range. The abnormal voice quality paper, we present a preliminary, online test of output severely affects the social life of people a real-time, interactive SSI based on electro- after laryngectomy (Liu and Ng, 2007). In addi- magnetic motion tracking. The SSI played tion, the tracheo-esophageal option requires an back synthesized speech sounds in response additional surgery, which is not suitable for eve- to the user’s tongue and lip movements.
  • Silent Speech Interfaces B

    Silent Speech Interfaces B

    Silent Speech Interfaces B. Denby, T. Schultz, K. Honda, Thomas Hueber, J.M. Gilbert, J.S. Brumberg To cite this version: B. Denby, T. Schultz, K. Honda, Thomas Hueber, J.M. Gilbert, et al.. Silent Speech Interfaces. Speech Communication, Elsevier : North-Holland, 2010, 52 (4), pp.270. 10.1016/j.specom.2009.08.002. hal- 00616227 HAL Id: hal-00616227 https://hal.archives-ouvertes.fr/hal-00616227 Submitted on 20 Aug 2011 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Accepted Manuscript Silent Speech Interfaces B. Denby, T. Schultz, K. Honda, T. Hueber, J.M. Gilbert, J.S. Brumberg PII: S0167-6393(09)00130-7 DOI: 10.1016/j.specom.2009.08.002 Reference: SPECOM 1827 To appear in: Speech Communication Received Date: 12 April 2009 Accepted Date: 20 August 2009 Please cite this article as: Denby, B., Schultz, T., Honda, K., Hueber, T., Gilbert, J.M., Brumberg, J.S., Silent Speech Interfaces, Speech Communication (2009), doi: 10.1016/j.specom.2009.08.002 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript.
  • Articulatory Information and Multiview Features for Large Vocabulary Continuous Speech Recognition

    Articulatory Information and Multiview Features for Large Vocabulary Continuous Speech Recognition

    ARTICULATORY INFORMATION AND MULTIVIEW FEATURES FOR LARGE VOCABULARY CONTINUOUS SPEECH RECOGNITION Vikramjit Mitra*, Wen Wang, Chris Bartels, Horacio Franco, Dimitra Vergyri University of Maryland, College Park, MD, USA Speech Technology and Research Laboratory, SRI International, Menlo Park, CA, USA [email protected], {wen.wang, chris.bartels, horacio.franco, dimitra.vergyri}@sri.com ABSTRACT acoustic conditions. Advanced variants of DNNs, such as convolutional neural nets (CNNs) [12], recurrent neural This paper explores the use of multi-view features and their nets (RNNs) [13], long short-term memory nets (LSTMs) discriminative transforms in a convolutional deep neural [14], time-delay neural nets (TDNNs) [15, 29], VGG-nets network (CNN) architecture for a continuous large [8], have significantly improved recognition performance, vocabulary speech recognition task. Mel-filterbank energies bringing them closer to human performance [9]. Both and perceptually motivated forced damped oscillator abundance of data and sophistication of deep learning coefficient (DOC) features are used after feature-space algorithms have symbiotically contributed to the maximum-likelihood linear regression (fMLLR) advancement of speech recognition performance. The role transforms, which are combined and fed as a multi-view of acoustic features has not been explored in comparable feature to a single CNN acoustic model. Use of multi-view detail, and their potential contribution to performance gains feature representation demonstrated significant reduction in is unknown. This paper focuses on acoustic features and word error rates (WERs) compared to the use of individual investigates how their selection improves recognition features by themselves. In addition, when articulatory performance using benchmark training datasets: information was used as an additional input to a fused deep Switchboard and Fisher, when evaluated on the NIST 2000 neural network (DNN) and CNN acoustic model, it was CTS test set [2].
  • Text-To-Speech Synthesis As an Alternative Communication Means

    Text-To-Speech Synthesis As an Alternative Communication Means

    Biomed Pap Med Fac Univ Palacky Olomouc Czech Repub. 2021 Jun; 165(2):192-197. Text-to-speech synthesis as an alternative communication means after total laryngectomy Barbora Repovaa#, Michal Zabrodskya#, Jan Plzaka, David Kalferta, Jindrich Matousekb, Jan Betkaa Aims. Total laryngectomy still plays an essential part in the treatment of laryngeal cancer and loss of voice is the most feared consequence of the surgery. Commonly used rehabilitation methods include esophageal voice, electrolarynx, and implantation of voice prosthesis. In this paper we focus on a new perspective of vocal rehabilitation utilizing al- ternative and augmentative communication (AAC) methods. Methods and Patients. 61 consecutive patients treated by means of total laryngectomy with or w/o voice prosthesis implantation were included in the study. All were offered voice banking and personalized speech synthesis (PSS). They had to voluntarily express their willingness to participate and to prove the ability to use modern electronic com- munication devices. Results. Of 30 patients fulfilling the study criteria, only 18 completed voice recording sufficient for voice reconstruction and synthesis. Eventually, only 7 patients started to use this AAC technology during the early postoperative period. The frequency and total usage time of the device gradually decreased. Currently, only 6 patients are active users of the technology. Conclusion. The influence of communication with the surrounding world on the quality of life of patients after total laryngectomy is unquestionable. The possibility of using the spoken word with the patient's personalized voice is an indisputable advantage. Such a form of voice rehabilitation should be offered to all patients who are deemed eligible.
  • Towards Robust Combined Deep Architecture for Speech Recognition : Experiments on TIMIT

    Towards Robust Combined Deep Architecture for Speech Recognition : Experiments on TIMIT

    (IJACSA) International Journal of Advanced Computer Science and Applications, Vol. 11, No. 4, 2020 Towards Robust Combined Deep Architecture for Speech Recognition : Experiments on TIMIT Hinda DRIDI1, Kais OUNI2 Research Laboratory Smart Electricity & ICT, SE&ICT Lab, LR18ES44 National Engineering School of Carthage, ENICarthage University of Carthage, Tunis, Tunisia Abstract—Over the last years, many researchers have getting ideal ASR systems. Thus, we should open the most engaged in improving accuracies on Automatic Speech relevant research topics to attain the principal goal of Recognition (ASR) task by using deep learning. In state-of-the- Automatic Speech Recognition. In fact, recognizing isolated art speech recognizers, both Long Short-Term Memory (LSTM) words is not a hard task, but recognizing continuous speech is a and Gated Recurrent Unit (GRU) based Reccurent Neural real challenge. Any ASR system has two parts: the language Network (RNN) have achieved improved performances model and the acoustic model. For small vocabularies, compared to Convolutional Neural Network (CNN) and Deep modeling acoustics of individual words is may be easily done Neural Network (DNN). Due to the strong complementarity of and we may get good speech recognition rates. However, for CNN, LSTM-RNN and DNN, they may be combined in one large vocabularies, developing successful ASR systems with architecture called Convolutional Long Short-Term Memory, good speech recognition rates has become an active issue for Deep Neural Network (CLDNN). Similarly we propose to combine CNN, GRU-RNN and DNN in a single deep architecture researchers. As vocabulary size grows, we will not be able to called Convolutional Gated Recurrent Unit, Deep Neural get enough spoken samples of all words and we should model Network (CGDNN).
  • Arxiv:1805.04699V4 [Cs.CL] 13 Jun 2019

    Arxiv:1805.04699V4 [Cs.CL] 13 Jun 2019

    TED-LIUM 3: Twice as Much Data and Corpus Repartition for Experiments on Speaker Adaptation Fran¸coisHernandez1, Vincent Nguyen1, Sahar Ghannay2, Natalia Tomashenko2, and Yannick Est`eve2 1 Ubiqus, Paris, France [email protected] https://www.ubiqus.com 2 LIUM, University of Le Mans, France [email protected] https://lium.univ-lemans.fr/ Abstract. In this paper, we present TED-LIUM release 3 corpus3 ded- icated to speech recognition in English, which multiplies the available data to train acoustic models in comparison with TED-LIUM 2, by a factor of more than two. We present the recent development on Auto- matic Speech Recognition (ASR) systems in comparison with the two previous releases of the TED-LIUM Corpus from 2012 and 2014. We demonstrate that, passing from 207 to 452 hours of transcribed speech training data is really more useful for end-to-end ASR systems than for HMM-based state-of-the-art ones. This is the case even if the HMM- based ASR system still outperforms the end-to-end ASR system when the size of audio training data is 452 hours, with a Word Error Rate (WER) of 6.7% and 13.7%, respectively. Finally, we propose two repar- titions of the TED-LIUM release 3 corpus: the legacy repartition that is the same as that existing in release 2, and a new repartition, calibrated and designed to make experiments on speaker adaptation. Similar to the two first releases, TED-LIUM 3 corpus will be freely available for the research community. Keywords: Speech recognition · Opensource corpus · Deep learning · Speaker adaptation · TED-LIUM.
  • A 8.93 TOPS/W LSTM Recurrent Neural Network Accelerator Featuring Hierarchical Coarse-Grain Sparsity with All Parameters Stored On-Chip

    A 8.93 TOPS/W LSTM Recurrent Neural Network Accelerator Featuring Hierarchical Coarse-Grain Sparsity with All Parameters Stored On-Chip

    A 8.93 TOPS/W LSTM Recurrent Neural Network Accelerator Featuring Hierarchical Coarse-Grain Sparsity with All Parameters Stored On-Chip Deepak Kadetotad, Visar Berisha, Chaitali Chakrabarti, Jae-sun Seo School of Electrical, Computer and Energy Engineering, Arizona State University, Tempe, AZ 85287, USA {deepak.kadetotad, visar, chaitali, jaesun.seo}@asu.edu Abstract —Long short-term memory (LSTM) networks are 1H[W 5HFXUUHQW /D\HU /D\HU widely used for speech applications but pose difficulties for KW /670 KW ࢏࢚ ൌ࣌ሺࢃ࢞࢏࢚࢞ ൅ࢃࢎ࢏ࢎ࢚െ૚ ൅࢈࢏ሻ efficient implementation on hardware due to large weight storage RW sig- E output gate • + R &HOO moid requirements. We present an energy-efficient LSTM recurrent ࢌ࢚ ൌ࣌ሺࢃ࢞ࢌ࢚࢞ ൅ࢃࢎࢌࢎ࢚െ૚ ൅࢈ࢌሻ neural network (RNN) accelerator, featuring an algorithm- [W tanh ࢕࢚ ൌ࣌ሺࢃ࢞࢕࢚࢞ ൅ࢃࢎ࢕ࢎ࢚െ૚ ൅࢈࢕ሻ KW forget gate hardware co-optimized memory compression technique called &W memory cell ̱ sig- • &W ࢉ࢚ ൌ ܜ܉ܖܐሺࢃ࢞ࢉ࢚࢞ ൅ࢃࢎࢉࢎ࢚െ૚ ൅࢈ࢉሻ EI + hierarchical coarse-grain sparsity (HCGS). Aided by HCGS- moid K + input gate W ̱ ࢉ࢚ ൌࢌ࢚ ήࢉ࢚െ૚ ൅࢏࢚ ήࢉ࢚ based block-wise recursive weight compression, we demonstrate [W sig- • LW EL moid + × 8QZHLJKWHG LSTM networks with up to 16 fewer weights while achieving a ࢎ࢚ ൌ࢕࢚ ήܜ܉ܖܐሺࢉ࢚ሻ &RQQHFWLRQ & W [W minimal accuracy loss. The prototype chip fabricated in 65nm LP :HLJKWHG candidate &RQQHFWLRQ tanh memory ࢎ࢚െ૚KLGGHQYHFWRU CMOS achieves 8.93/7.22 TOPS/W for 2-/3-layer LSTM RNNs &RQQHFWLRQ ࢞ LQSXWYHFWRU ZLWKWLPHODJ ࢚ trained with HCGS for TIMIT/TED-LIUM corpora. + ZHLJKWPDWULFHVכࢃࢎכࢃ࢞ KW [W Index Terms—long short-term memory, speech recognition, EF hardware accelerator, weight compression, structured sparsity Fig. 1. Illustration of LSTM cell with computation equations. weights by 16× with minimal accuracy loss.
  • Automatic Speech Recognition with Deep Neural Networks for Impaired Speech

    Automatic Speech Recognition with Deep Neural Networks for Impaired Speech

    Automatic Speech Recognition with Deep Neural Networks for Impaired Speech Cristina Espa~na-Bonet1;2 and Jos´eA. R. Fonollosa1 1 TALP Research Center, Universitat Polit`ecnicade Catalunya, Barcelona, Spain 2 Universit¨atdes Saarlandes, Saarbr¨ucken, Germany [email protected], [email protected] Abstract. Automatic Speech Recognition has reached almost human performance in some controlled scenarios. However, recognition of im- paired speech is a difficult task for two main reasons: data is (i) scarce and (ii) heterogeneous. In this work we train different architectures on a database of dysarthric speech. A comparison between architectures shows that, even with a small database, hybrid DNN-HMM models out- perform classical GMM-HMM according to word error rate measures. A DNN is able to improve the recognition word error rate a 13% for sub- jects with dysarthria with respect to the best classical architecture. This improvement is higher than the one given by other deep neural networks such as CNNs, TDNNs and LSTMs. All the experiments have been done with the Kaldi toolkit for speech recognition for which we have adapted several recipes to deal with dysarthric speech and work on the TORGO database. These recipes are publicly available. Keywords: speech recognition, speaker adaptation, deep learning, neu- ral networks, dysarthria, Kaldi 1 Introduction Automatic speech recognition (ASR) consists on automatically transcribing voice into text. It is not an easy task: one has do deal with noise, differences among speakers and spontaneous speech phenomena among others. For some controlled scenarios where one can minimise the effect of these phenomena, ASR approaches or exceeds the accuracy of humans on several benchmarks [2, 19].
  • Improving Speech Recognition by Revising Gated Recurrent Units

    Improving Speech Recognition by Revising Gated Recurrent Units

    Improving speech recognition by revising gated recurrent units Mirco Ravanelli1, Philemon Brakel2, Maurizio Omologo1, Yoshua Bengio2 1Fondazione Bruno Kessler, Trento, Italy 2Universite´ de Montreal,´ Montreal,´ Canada [email protected], [email protected], [email protected], [email protected] Abstract Markov Models (HMMs) in DNN/HMM speech recognizers. Training RNNs, however, can be complicated by vanish- Speech recognition is largely taking advantage of deep learning, ing and exploding gradients, which might impair learning long- showing that substantial benefits can be obtained by modern term dependencies [22]. Although exploding gradients can ef- Recurrent Neural Networks (RNNs). The most popular RNNs fectively be tackled with simple clipping strategies [23], the are Long Short-Term Memory (LSTMs), which typically reach vanishing gradient problem requires special architectures to be state-of-the-art performance in many tasks thanks to their abil- properly addressed. A common approach relies on the so- ity to learn long-term dependencies and robustness to vanishing called gated RNNs, whose core idea is to introduce a gating gradients. Nevertheless, LSTMs have a rather complex design mechanism for better controlling the flow of the information with three multiplicative gates, that might impair their efficient through the various time-steps. Within this family of archi- implementation. An attempt to simplify LSTMs has recently tectures, vanishing gradient issues are mitigated by creating ef- led to Gated Recurrent Units (GRUs), which are based on just fective “shortcuts”, in which the gradients can bypass multiple two multiplicative gates. temporal steps. This paper builds on these efforts by further revising GRUs and proposing a simplified architecture potentially more suit- The most popular gated RNNs are Long Short-Term Mem- able for speech recognition.
  • Speaker-Aware Training of Lstm-Rnns for Acoustic Modelling

    Speaker-Aware Training of Lstm-Rnns for Acoustic Modelling

    SPEAKER-AWARE TRAINING OF LSTM-RNNS FOR ACOUSTIC MODELLING Tian Tan 1, Yanmin Qian1;2, Dong Yu3, Souvik Kundu4, Liang Lu5, Khe Chai SIM4, Xiong Xiao6, Yu Zhang7 1Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai, China 2 Cambridge University Engineering Department, Cambridge, UK 3Microsoft Research, Redmond, USA 4School of Computing, National University of Singapore, Republic of Singapore 5Centre for Speech Technology Research, University of Edinburgh, UK 6Temasek Labs, Nanyang Technological University, Singapore. 7CSAIL, MIT, Cambridge, USA ABSTRACT this context, Long Short-Term Memory (LSTM) [4] is proposed to overcome these problems by introducing separate gate functions to Long Short-Term Memory (LSTM) is a particular type of recurrent control the flow of the information. For acoustic modelling, LSTM- neural network (RNN) that can model long term temporal dynam- RNNs are reported to outperform the DNNs on the large vocabulary ics. Recently it has been shown that LSTM-RNNs can achieve tasks [5, 6]. higher recognition accuracy than deep feed-forword neural net- However, a long standing problem in acoustic modelling is the works (DNNs) in acoustic modelling. However, speaker adaption mismatch between training and test data caused by speakers and en- for LSTM-RNN based acoustic models has not been well inves- vironmental differences. Although DNNs are more robust to the mis- tigated. In this paper, we study the LSTM-RNN speaker-aware match compared to GMMs, significant performance degradation has training that incorporates the speaker information during model been observed [7]. There have been a number of studies to improve training to normalise the speaker variability.
  • UNIVERSITY of CALIFORNIA Los Angeles Neural Network Based

    UNIVERSITY of CALIFORNIA Los Angeles Neural Network Based

    UNIVERSITY OF CALIFORNIA Los Angeles Neural network based representation learning and modeling for speech and speaker recognition A dissertation submitted in partial satisfaction of the requirements for the degree Doctor of Philosophy in Electrical and Computer Engineering by Jinxi Guo 2019 c Copyright by Jinxi Guo 2019 ABSTRACT OF THE DISSERTATION Neural network based representation learning and modeling for speech and speaker recognition by Jinxi Guo Doctor of Philosophy in Electrical and Computer Engineering University of California, Los Angeles, 2019 Professor Abeer A. H. Alwan, Chair Deep learning and neural network research has grown significantly in the fields of automatic speech recognition (ASR) and speaker recognition. Compared to traditional methods, deep learning-based approaches are more powerful in learning representation from data and build- ing complex models. In this dissertation, we focus on representation learning and modeling using neural network-based approaches for speech and speaker recognition. In the first part of the dissertation, we present two novel neural network-based methods to learn speaker-specific and phoneme-invariant features for short-utterance speaker verifi- cation. We first propose to learn a spectral feature mapping from each speech signal to the corresponding subglottal acoustic signal which has less phoneme variation, using deep neural networks (DNNs). The estimated subglottal features show better speaker-separation abil- ity and provide complementary information when combined with traditional speech features on speaker verification tasks. Additional, we propose another DNN-based mapping model, which maps the speaker representation extracted from short utterances to the speaker rep- resentation extracted from long utterances of the same speaker. Two non-linear regression models using an autoencoder are proposed to learn this mapping, and they both improve speaker verification performance significantly.