Paolo Baggia Loquendo Speech Technologies and Standards DIT Seminars, Povo, Trento June 7th, 2006 © 2006 Loquendo SpA Loquendo Today Global company formed in 2001 as a spin-off from the Telecom Italia R&D center with over 30 years experience in Speech Technologies A Telecom Italia Group company HQ in Turin (Italy) and Sales offices in Miami (US), Madrid (Spain), Munich (Germany) and Paris (France), and a Worldwide Network of Agents. Market-leader in Italy, Spain, Greece with a strong presence in Europe, growing presence in North and Latin America Strategy:Strategy: “Best Innovation ¾¾ CompleteComplete set set of of speech speech technologies technologies on on aa wide wide spectrum spectrum of of devices devices in Multi-Lingual Speech Synthesis” Prize ¾¾ ReadyReady for for challenging challenging future future scenarios: scenarios: Multimodality,Multimodality, Security Security AVIOS-SpeechTEK West 2006 ¾¾ PartnershipPartnership as as a a key key factor factor © 2006 Loquendo SpA 2 Plan of my talk • The Past – Evolution of Speech Technologies (mainly ASR, TTS) – First Speech Applications (Touch Tones, IVRs) • The Present – The Impact of Standards – The (r)evolution of VoiceXML – A Constellation of W3C Standards – Work in progress • The Future – Voice and Video Applications – Speech and SemanticWeb – Towards Multimodality • Q&A © 2006 Loquendo SpA 3 A Brief History of Spoken Language Technology BL's Voder Von Kempelen's Dictation systems Spoken dialog Talking Machine Dictation Industry Industry Radio Rex Widespread use of MultiModal BL's Audrey HMMs NLU systems Conversational systems 1769 1920 1952 19711975 1982 1988 1995 2000 1936 1985 1990 FFT, cepstrum, Dialog DTW, TTS DARPA Isolated Words COMMUNICATOR Continuous DARPA SUR Speech Speech program starts DARPA Recognition Understanding Hidden Markov Resource DARPA ATIS Models Management © 2006 Loquendo SpA by Roberto Pieraccini 4 Evolution of Speech & Language • Foundations lay in many fields: – Computer science, electronic engineering, linguistics and psychology • ’50 – ’70 : Two paradigms – Symbolic: Formal syntax, AI techniques – Stochastic: Bayesian method, corpora (es. Brown Corpus) • ‘70 – mid ’80 : Four separate fields – Stochastic: • Dynamic Programming, Viterbi algorithm • hidden Markov models • Back-forward algorithm for training HMMs • Language Modeling (bigrams, trigrams, smoothing techniques) – Logic-based: DFG, functional grammars, LFG – Natural Language Understanding: Winograd, Woods, many others – Discourse Modeling: Grosz, Sidner, Allen • mid ’80 - ’90 : Return of empiricism – Return to finite state modeling: AT&T Bell Labs – Rise of probabilistic models: IBM J. Watson research center • mid ’90 – now : The fields come together – The separation is partially blurred – Probabilistic and corpus driven in natural language processing © 2006 Loquendo SpA 5 Evolution of Speech Synthesis • Early days: – From: von Kempelen’s Talking Machine (1791) – Sir Charles Wheatstone (1835), Faber’s Euphonia (1835), etc. – To: Dudley’s Voder at NY World Fair 1939 • ’50 – ’60: – Audio Spectrograms, first articulatory and formant approaches • mid ’70 – ‘90: – KlattTalk, MIT Talk, DECTalk – Kurzweil Machine for the visually impaired – IBM and AT&T progresses Î Very intelligible TTS, but still robotic • mid ’90: – Concatenation by Unit Selection Î Intelligible and very natural voices • Today – Lots of application fields for TTS: SF-embedded, mobile devices, navigation systems, IVRs (!) Î but also search engines, domotic, multimodal, etc. © 2006 Loquendo SpA 6 R&D in Italy • First attempts: Luigi Stringa, ELSAG • R&D: – Olivetti (Speech lab in Turin, closed in 1990) • Speech dictation, Speech synthesis (VoxPC) – IRST (lab in Povo, Trento, by Istituto Trentino di Cultura) • Speech recognition, dictation, speech synthesis, spoken dialog – CSELT, then TILAB • Speech recognition, Speech synthesis, Speaker Verification • Speech recognition on chip • First large speech applications deployed Servizio 1412 e 12 (Telecomitalia), FS_INFORMA (Trenitalia) • Speech Technologies and Platforms: – LOQUENDO (spin-off in 2001) • Engines: speech recognition, speech synthesis, speaker verification • Embedded speech technologies • VoiceXML platform (VoxNauta) and MRCP-based server (Loquendo Speech Suite) © 2006 Loquendo SpA 7 Introduction to Speech Technologies • ASR: Automatic Speech Recognition • TTS: Text-To-Speech • SIV: Speaker Identification Verification • Embedded Speech Technologies © 2006 Loquendo SpA Introduction to Speech Technologies • Goal: To extract information from a speech signal Automatic Spoken Speech Meaning: Words: Understanding [Day: tomorrow Recognition Tomorrow in Part_of_Day: morning ] Acoustic the morning Signal Language Language: Identification Italian Speaker Speaker: Identification Baggia Paolo • Goal: To generate a speech signal from content Acoustic Signal Text-To-Speech Generation Meaning: Text: [ … ] ……. © 2006 Loquendo SpA 9 ASR: Automatic Speech Recognition © 2006 Loquendo SpA 10 ASR block diagram Vocal (X) (Ŵ)(Ŝ) signal Acoustical Phonetic Linguistic Analyzer Decoder Interpretation (Front-End) (Search) Linguistic and Statistical models domain of phonetic events information (X) Parametric description of the voice signal (Ŵ) Sequence of recognized words (Ŝ) Semantic representation © 2006 Loquendo SpA 11 Automatic speech recognizers: main features Features Training mode Focused on a Speaker unique speaker independent Pronunciation Isolated Continuous mode Environment Controlled Difficult condition Recognition Constrained Customizable Dictation Domain • Current telephonic recognizers are speaker independent, supporting continuous speech input; the recognition domain must be defined by the application developer • The dictation on a general domain is today available with speaker adapted desktop recognizers: • The recognizer models are focused on the target speaker voice • The environment must be good (high quality microphone, low noise) • However, the domain must not be unconstrained … to obtain high performance specific additional information should be supplied © 2006 Loquendo SpA 12 HMM-NN Stationary - Transition Units Emission probability Hybrid Output Layer Models Second hidden Layer First hidden Innovations Layer • Multi-source NN Additive Features • MLP topology Base Features Alternative Features left context center right context • Acceleration method Input Layer . Base Feature: MFCC Signal Frames Time BASEADDITIVE ALTERNATIVE Feature Extraction Alternative Features : FEATURES FEATURES FEATURES Modules Rasta-PLP, Ear-Model Additive Features : Gravity Centers, Frequency Derivatives Acoustic Signal © 2006 Loquendo SpA 13 ASR Typical Performance • The accuracy and efficiency performance on a vocabulary depends on the vocabulary size and confusability Telephonic environment 98.5 ÷ 99% digits (PSTN DB) 95 ÷ 97% 500 words vocabulary (Railways Stations: FS-Informa) 85 ÷ 90% 10,000 words vocabulary (Italian cities: Telecom Italia 12) • Kind of applications: – Multiple choice menus – Large vocabulary recognition (e.g Telecom Italia 12) © 2006 Loquendo SpA 14 ASR performance (by D. Pallet, NIST) WER years © 2006 Loquendo SpA 15 Hot Issues in ASR • Robustness – to the environment: noise (in-car, in house), side speech (mobile) – to the channel: VoIP channel, mobile channel – to the speaker: individual peculiarities (adaptation techniques) • Spontaneous Speech – Interruptions, restarts, extralinguistic phenomena • Speech search, audio indexing • Speech for very large applications – Integration of many knowledge sources © 2006 Loquendo SpA 16 TTS: Text-To-Speech © 2006 Loquendo SpA 17 Evolution of TTS quality Acoustic Loquendo TTS Quality Human voice 5 Loquendo TTS 4 3 ELOQUENS Timber Intelligibility 2 Loquendo TTS natural excellent 1 MUSA ELOQUENS acceptable good MUSA robotic sufficent 1st 2nd 3rd generation © 2006 Loquendo SpA 18 Three Families of Speech Synthesizers • Articulatory synthesizers • Formant synthesizers • Concatenative synthesizers – Diphone based (concatenation of small units) – Unit Selection (concatenation of larger units) D. Klatt, JASA, vol. 82 no. 3, Sept. 1987 © 2006 Loquendo SpA 19 Corpus based, Unit Selection TTS Large corpora Input text Linguistic that statistically Knowledge represent the language ITALIAN ENGLISH Internal tools for: Textual Analysis • DB design ... • Signal acquisition • Phoneme Segmentation Phoneme Labels & • Signal Analysis prosody markers ITALIAN Voice 1 Acoustic ITALIAN Output Voice 2 ENGLISH Loquendo TTS Acoustic Dictionaries Large DB: Speech Signal 200 Mb per voice © 2006 Loquendo SpA 20 Loquendo TTS Features • Interactive demo of the 39 voices in 17 languages http://www.loquendo.com/en/demos/demo_tts.htm • Innovative features released in the TTS engine: – Expressiveness Conventional figures, different styles, expressive cues – Customizable voices Change in timbre, save and reuse a customized voice – Mixing TTS and music Music and TTS are synchronized SMSSMS reading reading rr u u don don nethng nethng 2mor? 2mor? would would b b gr8gr8 2cu. 2cu. i'lli'll b b @ @ pub pub b4 b4 8. 8. ringl8r.ringl8r. b4n b4n xoxox xoxox susan susan © 2006 Loquendo SpA 21 Mixed Language Capability Loquendo TTS automatically detects each change of language and gives the user the option to: Change voice, choosing from the voices available, based on the language detected. Keep the same voice as the original language, and apply phonetic mapping between the newly detected language and the original one. Many movies have been produced and filmed in