Linguistic processing for Spoken Language Understanding Fr´ed´ericB´echet Aix Marseille Universit´e- Laboratoire d'Informatique Fondamentale - LIF-CNRS December 18th 2014 F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 1 / 66 Spoken Language Understanding 1 Introduction 2 Parsing Speech 3 Dependency parsing for Open Domain SLU 4 Speech-to-Speech translation F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 2 / 66 Spoken Language Understanding Parsing speech for Spoken Language Understanding (SLU) of spontaneous speech Spoken Language Understanding Natural Language Automatic Speech Processing Recognition Machine Learning F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 3 / 66 Spoken Language Understanding -Automatic Speech Recognition transcription ● Transcription ● « Rich »Transcription -Information Extraction -Analysis ● classification Signal Processing ● Syntactic, semantic ● Extracting entities understanding ● Relations between entities ● …. F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 4 / 66 Two applicative frameworks Human-Machine spoken dialogue I Call-routing I Form-filling I Personal assistant I Computer assisted task Speech Analytics / Voice Search I Broadcast News I Large speech archives I Call Centres Main challenge : processing spontaneous speech I Human/Machine conversation I Human/Human conversation F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 5 / 66 Choice of a Meaning Representation Language General view I meaning as the composition of basic constituents I Definition of constituents and relations independent from a given application Application specific view I interpretation = A representation that can be executed by an interpreter in order to change the state of the system (Speech Communication 48 - SLU) I Goal of SLU from a system point of view ! SYSTEM INTERPRETATION F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 6 / 66 Choice of a Meaning Representation Language Can be defined by the application framework I In a call-routing application ! Call-type I In a database query application ! SQL query I In a directory assistance application ! entry in the directory I In a speech analytics / voice search application F Distillation / Summarization F Behavioral analysis F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 7 / 66 Choice of a Meaning Representation Language Can be a formal language I Flat representation ! concepts F Named entities, Sequences of keywords, Verbs, ... I Structured representation F Logical formulae F predicate/argument structure (FrameNet) + Dialog act + reference resolution F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 8 / 66 Which models and algorithm to perform SLU ? Different points of view I Understanding is a classification task ! I Understanding is a translation task ! I Understanding is a parsing task ! F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 9 / 66 Understanding is a classification task Mapping a speech message to a label I Call-type (How May I Help You ?), dialog act (CALO), user intent (personal assistant) . Direct system interpretation of a message Classifiers I Support Vector Machines, Boosting, . F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 10 / 66 Understanding is a translation task Translation to a formal semantic language I Related to tagging approaches (e.g. POS tagging) Main model : Concept decoding I Mapping a sequence of words to a sequence of attribute/value tokens Main approaches I Hidden Markov Models, MaxEnt tagger, Conditional Random Fields F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 11 / 66 Understanding is a parsing task Syntactic/Semantic parsing Structured representation I Constituent tree, dependency tree F System intepretation F obtained directly from the parse tree Examples I Full parse : mapping syntactic trees to semantic trees F Deep Understanding (Allen, ACL 2007) I Shallow parsing F Robust parsing TINA (MIT) I Parsing + Semantic r^olelabelling F SLU with parsing + SVM classifiers (Moschitti, ASRU 2007) F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 12 / 66 Complex Understanding Human-Human Conversation Analysis - Good morning, - Goodbye. how may I help you? - No, that's fine, goodbye. - Hello. I am calling regarding my internet connection. - Do you need anything else? Opening Closing - What's your phone number - Oh great, thanks. please? Information Verification Problem - Alright, it doesn't work - It's 555-221-5282. Resolution anymore. We will send you a new one shortly. - I see, you have a connection problem, right? Conflict Situtation - Hmm... - Yes, it's the fourth time I call about that. - Sir, please calm down. Let me remotely check your modem. - Okay, can you check that the light is... - I already did all the checks last time! Can't you just send a repairman? Relevance Segment Meaning Emotion Dialog is Conflict about procedure[state=completed,time=past] problematic procedure query[obj=intervention,loc=home] Behaviour Analysis Answer complex questions - Was the operator empathic to customer's concerns? about conversations: - Was the customer satistified ? - How does the agent manage the argument? - Did the agent achieve the goal of the call? - Is the agent polite? F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 13 / 66 Spoken Language Understanding 1 Introduction 2 Parsing Speech 3 Dependency parsing for Open Domain SLU 4 Speech-to-Speech translation F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 14 / 66 Why is syntactic parsing needed for SLU ? Syntactic relations ! semantic relations I From syntactic dependency to semantic dependency I Ex : The CoNLL-2008 Shared Task on Joint Parsing of Syntactic and Semantic Dependencies Semantic disambiguisation I By identifying the arguments of a verb and using semantic constraints on the arguments Ex : Disambiguisation between Organization / Location entities France has made a statement about .. / I'll visit France next year Identifying the language register I Read speech / prepared speech / spontaneous speech I Speaker role identification F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 15 / 66 Speech is not text Text I Complete object : made to be read ! ! I Structured object I Designed according to strong specifications Speech I Dynamic object : edit, repairs, repetition, hesitations I Structure = mostly at the acoustic level I Speaking style : read/prepared/spontaneous, dialogs F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 16 / 66 ASR output is not text ! ! ASR output I Partial information : all the acoustic dimension of speech is missing I Stream of words, no punctuations ASR errors I Insertion/deletion/substitution I Each word can be weighted by a confidence score F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 17 / 66 Non-native text Generation of a transcription W from a signal A : I P (AjW ) = acoustic models I P (W ) = language models ! text generation model W^ = max P (AjW ) × P (W ) W Therefore ! closed vocabulary I All words are included in the generative model P (W ) I Automatic transcriptions contain no unknown words ! ! Sentence segmentation I pause, fixed duration, a posteriori process based on prosodic/lexical features Disfluencies ! ASR errors Multiple hypotheses with confidence scores I Word lattices, word confusion networks, n-best lists F. B´echet (AMU LIF-CNRS) Xerox Research Centre Europe, Grenoble December 18th 2014 18 / 66 Transcribing spontaneous speech euh bonjour donc c' est XX à l' appareil je sais pas si vous savez très bien qui je suis euh donc par rapport au à la niveau de la satisfaction de ma satisfaction personnelle par rapport à votre service euh je dirai que dans l' ensemble je suis euh plutôt satisfait euh vous avez un très bon service clientèle qui sait écouter qui euh non qui j' ai pas grand chose à dire c' est c' est très très bien sinon ben juste par rapport au à ce que vous avez mis en place euh tout de suite justement c' est une très bonne idée justement d' une façon à ce qu' y ait un taux de réponse euh assez important maintenant c' est vrai qu' on est obligé de rappeler plusieurs fois et encore quand on prend le temps de rappeler pour euh pour euh pour euh pour répondre parce que quand on nous dit euh vous allez nous donner vous allez donner euh votre euh vos idées euh vos vos suggestions et ben on n' a rien en tête donc c' est pour ça que j' ai été obligé de raccrocher et de réfléchir à ce que je vais vous dire on ça c' est pas je pense euh que ça c' est le point collectif ou c' est le point négatif et sinon dans l' ensemble je suis très satisfait sinon y a une chose que j' ai à noter euh j' ai deux comptes chez vous euh je trouve ça un peu embêtant de pouvoir euh de pas pouvoir accéder euh aux deux par la même personne quand j' appelle mon service clientèle donc ça je trouve ça un peu dommage que je sois obligé de dépenser en plus parce que faut que je c' est pas le même type euh c' est pas la même personne qui s' occupe de mon dossier donc ce qui aurait été bien c' est quand même regrouper les deux dossiers sous euh euh sous un seul quoi de façon à ce que quand on appelle on puisse accéder aux deux dossiers séparément bien sûr mais les deux dossiers donc voilà euh sinon ben je vous remercie en tous cas pour euh pour votre gentillesse et votre amabilité vos conseillers clientèle sont très très gentils et très à l' écoute et donc je vous en remercie au revoir bonne journée bonne soirée F.