Domain Specific Acts for Spoken Translation

Lori Levin, Chad Langley, Alon Lavie, Donna Gates, Dorcas Wallace and Kay Peterson Language Technologies Institute Carnegie Mellon University Pittsburgh, PA, United States {lsl,clangley,alavie,dmg,dorcas,kay+}@cs.cmu.edu

main specific concepts. Examples of domain ac- Abstract tions and speech acts are shown in Figure 1.

We describe a coding scheme for machine c:give-information+party “I will be traveling with my husband and translation of spoken task-oriented dia- our two children ages two and eleven” logue. The coding scheme covers two − c:request-information+existence+facility levels of speaker intention domain in- “Do they have parking available?" dependent speech acts and domain de- “Is there someplace to go ice skating?" pendent domain actions. Our database c:give-information+view+information-object contains over 14,000 tagged sentences in “I see the bus icon” English, Italian, and German. We argue that domain actions, and not speech acts, Figure 1: Examples of Speech Acts and Domain are the relevant discourse unit for improv- Actions. ing translation quality. We also show that, although domain actions are domain spe- Domain actions are constructed compositionally cific, the approach scales up to large do- from an inventory of speech acts and an inventory mains without an explosion of domain of concepts. The allowable combinations of speech actions and can be coded with high inter- acts and concepts are formalized in a human- and coder reliability across research sites. Fur- machine-readable specification document. The thermore, although the number of domain specification document is supported by a database actions is on the order of ten times the of over 14,000 tagged sentences in English, Ger- number of speech acts, sparseness is not a man, and Italian. problem for the training of classifiers for The discourse community has long recognized identifying the domain action. We de- the potential for improving NLP systems by identi- scribe our work on developing high accu- fying speaker intention. It has been hypothesized racy speech act and domain action that predicting speaker intention of the next utter- classifiers, which is the core of the source ance would improve speech recognition (Reith- language analysis module of our inger et al., Stolcke et al.), or reduce ambiguity for NESPOLE machine translation system. machine translation (Qu et al., 1996, Qu et al., 1997). Identifying speaker intention is also critical for sentence generation. 1 Introduction We argue in this paper that the explicit repre- sentation of speaker intention using domain actions The NESPOLE and C-STAR machine translation can serve as the basis for an effective language- projects use an interlingua representation based on independent representation of meaning for speech- speaker intention rather than literal meaning. The to-speech translation and that the relevant units of speaker's intention is represented as a domain in- speaker intention are the domain specific domain dependent speech act followed by domain depend- action as well as the domain independent speech ent concepts. We use the term domain action to act. After a brief description of our database, we refer to the combination of a speech act with do- present linguistic motivation for domain actions. summarizes the number of tagged SDUs in com- We go on to show that although domain actions are plete dialogues in the interlingua database. There domain specific, there is not an explosion or expo- are some additional tagged dialogue fragments that nential growth of domain actions when we scale up are not counted. Figure 2 shows an excerpt from to a larger domain or port to a new domain. Finally the database. we will show that, although the number of domain actions is on the order of ten times the number of English NESPOLE Travel 4691 speech acts, data sparseness is not a problem in English C-STAR Travel 2025 training a domain action classifier. We present ex- German NESPOLE Travel 1538 tensive work on developing a high-accuracy classi- Italian NESPOLE Travel 2248 fier for domain actions using a variety of English Medical Assistance 2001 classification approaches and conclusions on the German Medical Assistance 1152 adequacy of these approaches to the task of domain Italian Medical Assistance 935 action classification. Table 1: Tagged SDUs in the Interlingua Database.

2 Data Collection Scenario and Database e709wa.19.0 comments: DATA from e709_1_0018_ITAGOR_00 Our study is based on data that was collected for the NESPOLE and C-STAR speech-to-speech e709wa.19.1 olang ITA lang ITA Prv CMU “hai in mente una localita specifica?" translation projects. Three domains are included. e709wa.19.1 olang ITA lang GER Prv CMU The NESPOLE travel domain covers inquiries “haben Sie einen bestimmten Ort im Sinn?" e709wa.19.1 olang ITA lang FRE Prv CLIPS about vacation packages. The C-STAR travel do- “" main consists largely of reservation and payment e709wa.19.1 olang ITA lang ENG Prv CMU dialogues and overlaps only about 50% in vocabu- “do you have a specific place in mind" e709wa.19.1 IF Prv CMU lary with the NESPOLE travel domain. The medi- a:request-information+disposition+object cal assistance domain includes dialogues about (object-spec=(place, modifier=specific, identifiability=no), disposi- chest pain and flu-like symptoms. tion=(intention, who=you)) There were two data collection protocols for the e709wa.19.1 comments: Tagged by dmg NESPOLE travel domain − monolingual and bilin- gual. In the monolingual protocol, an English Figure 2: Excerpt from the Interlingua Database. speaker in the United States had a conversation with an Italian travel agent speaking (non-native) 3 Linguistic Argument for Domain Ac- English in Italy. Monolingual data was also col- tions lected for German, French and Italian. Bilingual data was collected during user studies with, for Proponents of Construction (Fillmore et. example, an English speaker in the United States al. 1988, Goldberg 1995) have argued that human talking to an Italian-speaking travel agent in Italy, consist of constructional units that in- with the NESPOLE system providing the transla- clude a syntactic structure along with its associated tion between the two parties. The C-STAR data semantics and pragmatics. Some constructions fol- consists of only monolingual role-playing dia- low the typical syntactic rules of the language but logues with both speakers at the same site. The have a semantic or pragmatic focus that is not medical dialogues are monolingual with doctors compositionally predictable from the parts. Other playing the parts of both doctor and patient. constructions do not even follow the typical syntax The dialogues were transcribed and multi- of the language (e.g., Why not go? with no tensed sentence utterances were broken down into multi- verb). ple Semantic Dialogue Units (SDUs) that each cor- Our work with multilingual machine translation respond to one domain action. Some SDUs have of spoken language shows that fixed expressions been translated into other NESPOLE or C-STAR cannot be translated literally. For example, Why languages. Over 14,000 SDUs have been tagged not go to the meeting? can be translated into Japa- with interlingua representations including domain nese as Kaigi ni itte mitara doo? (meeting to going actions as well as argument-value pairs. Table 1 see/try-if how), which differs from the English in several ways. It does not have a correspond- ing to not; it has a word that means see/try that Travel Medical Combined does not appear in the English sentence; and so on. DAs 880 459 1168 In order to produce an acceptable translation, we SAs 67 44 72 must find a common ground between the English Concepts 91 74 125 fixed expression Why not V-inf? and the Japanese Table 2: DA component counts in NESPOLE data. fixed expression -te mittara doo?. The common ground is the speaker's intention (in this case, to Our domain action based interlingua has quite high make a suggestion) rather than the syntax or literal coverage of the travel and medical dialogues we meaning. have collected. To measure how well the interlin- Speaker intention is partially captured with a di- gua covers a domain, we define the no-tag rate as rect or indirect speech act. However, whereas the percent of sentences that are not covered by the speech acts are generally domain independent, interlingua, according to a human expert. The no- task-oriented language abounds with fixed expres- tag rate for the English NESPOLE travel dialogues sions that have domain specific functions. For ex- is 4.3% for dialogues that have been used for sys- ample, the phrases We have… or There are… in the tem development. hotel reservation domain express availability of We have also estimated the domain action no- rooms in addition to their more literal meanings of tag rate for unseen data using the NESPOLE travel possession and existence. In the past six years, we database (English, German, and Italian combined). have been successful in using domain specific do- We randomly selected 100 SDUs as seen data and main actions as the basis for translation of limited- extracted their domain actions. We then randomly domain task-oriented spoken language (Levin et selected 100 additional SDUs from the remaining al., 1998, Levin et al. 2002; Langley and Lavie, data and estimated the no-tag rate by counting the 2003) number of SDUs not covered by the domain ac- tions in the seen data. We then added the unseen 4 Scalability and Portability of Domain data to the seen data set and randomly selected 100 Actions new SDUs. We repeated this process until the en- tire database had been seen, and we repeated the Domain actions, like speech acts, convey speaker entire sampling process 10 times. Although the intention. However, domain actions also represent number of domain actions increases steadily with components of meaning and are therefore more the database size (Figure 4), the no-tag rate for un- numerous than domain independent speech acts. seen data stabilizes at less than 10%. 1168 unique domain actions are used in our We also randomly selected half of the SDUs NESPOLE database, in contrast to only 72 speech (4200) from the database as seen data and ex- acts. We show in this section that domain actions tracted the domain actions. Holding the seen data yield good coverage of task-oriented domains, that set fixed, we then estimated the no-tag rates in in- domain actions can be coded effectively by hu- creasing amounts of unseen data from the remain- mans, and that scaling up to larger domains or ing half of the database. We repeated this process porting to new domains is feasible without an ex- 10 times. With a fixed amount of seen data, the no- plosion of domain actions. tag rate remains stable for increasing amounts of

unseen data. We observed similar no-tag rate re- Coverage of Task-Oriented Domains: Our sults for the medical assistance domain and for the NESPOLE domain action database contains dia- combination of travel and medical domains. logues from two task-oriented domains: medical It is also important to note that although there is assistance and travel. Table 2 shows the number of a large set of uncommon domain actions, the top speech acts and concepts that are used in the travel 105 domain actions cover 80% of the sentences in and medical domains. The 1168 unique domain the travel domain database. Thus domain actions actions that appear in our database are composed are practical for covering task-oriented domains. of the 72 speech acts and 125 concepts.

Intercoder Agreement: Intercoder agreement is well as the growth in domain actions for a database another indicator of manageability of the domain consisting of equal amounts of data from each do- action based interlingua. We calculate intercoder main. Figure 4 shows the growth in domain actions agreement as percent agreement. Three interlingua for combined English, German, and Italian data in experts at one NESPOLE site achieved 94% the NESPOLE travel and medical domains. agreement (average pairwise agreement) on speech Figure 3 and Figure 4 show that the number of acts and 88% agreement on domain actions. Across domain actions increases steadily as the database sites, expert agreement on speech acts is still quite grows. However, closer examination reveals that high (89%), although agreement on domain actions scalability to larger domains and portability to new is lower (62%). Since many domain actions are domains are in fact feasible. The curves represent- similar in meaning, some disagreement can be tol- ing combined domains (travel plus medical in erated without affecting translation quality. Figure 4 and NESPOLE travel, C-STAR travel, and medical in Figure 3) show only a small in-

900 crease in the number of domain actions when two

800 domains are combined. In fact, there is a large s e l p

m 700 a overlap between domains. In Table 3 the Overlap S

m

o 600 d columns show the number of DA types and tokens n a R

0 500

1 that are shared between the travel and medical do-

r e v o

400

s mains. We can see around 70% of DA tokens are A D

e 300 u covered by DA types that occur in both domains. q i n U

200 n

a e M 100 DA Type DA Token 0 0 1000 2000 3000 4000 5000 6000 7000 Types Overlap Tokens Overlap SDUs per Sample NESPOLE 6004 Nespole Travel Nespole Medical C-STAR 880 171 8477 Nespole Travel+Medical C-STAR + Nespole Travel+Medical Travel (70.8%) NESPOLE 2743 Figure 3: DAs to cover data (English). 459 171 4088 Medical (67.1%)

1000 Table 3: DA Overlap (All languages).

900 s e l 800 p 5 A Hybrid Analysis Approach for Pars- m a S 700 m

o ing Domain Actions d n

a 600 R

0 1

r

e 500 Langley et al. (2002; Langley and Lavie, 2003) v o

s

A 400

D describe the hybrid analysis approach that is used

e u q i 300 n in the NESPOLE! system (Lavie et al., 2002). The U

n a

e 200

M hybrid analysis approach combines grammar-based 100 phrasal parsing and machine learning techniques to 0 0 1000 2000 3000 4000 5000 6000 7000 8000 9000 transform utterances into our interlingua represen- SDUs per Sample tation. Our analyzer operates in three stages to Nespole Travel Nespole Medical Nespole Travel+Medical identify the domain action and arguments. Figure 4: DAs to cover data (All languages). First, an input utterance is parsed into a se- quence of arguments using phrase-level semantic Scalability and Portability: The graphs in Figure 3 and the SOUP parser (Gavaldà, 2000). and Figure 4 illustrate growth in the number of Four grammars are defined for argument parsing: domain actions as the database size increases and an argument grammar, a pseudo-argument gram- as new domains are added. The x-axis represents mar, a cross-domain grammar, and a shared gram- the sample size randomly selected from the data- mar. The argument grammar contains phrase-level base. The y-axis shows the number of unique do- rules for parsing arguments defined in the interlin- main actions (types) averaged over 10 samples of gua. The pseudo-argument grammar contains rules each size. Figure 3 shows the growth in domain for parsing common phrases that are not covered actions for three English databases (NESPOLE by interlingua arguments. For example, all booked travel, C-STAR travel, and medical assistance) as up, full, and sold out might be grouped into a class ent approaches and/or feature sets for each task, of phrases that indicate unavailability. The cross- and reduces data sparseness. Our hybrid analyzer domain grammar contains rules for parsing com- uses the output of each classifier along with the plete DAs that are domain independent. For exam- interlingua specification to identify the DA (Lang- ple, this grammar contains rules for greetings ley et al., 2002; Langley and Lavie, 2003). (Hello, Good bye, Nice to meet you, etc.). Finally, the shared grammar contains low-level rules that 7 Experimental Setup can be used by all other subgrammars. After argument parsing, the utterance is seg- We conducted experiments to assess the perform- mented into SDUs using memory-based learning ance of several machine-learning approaches on (k-nearest neighbor) techniques. Spoken utterances the DA classification tasks. We evaluated all of the often consist of several SDUs. Since DAs are as- classifiers on English and German input in the signed at the SDU level, it is necessary to segment NESPOLE travel domain. utterances before assigning DAs. Corpus The final stage in the hybrid analysis approach 7.1 is domain action classification. The corpus used in all of the experiments was the NESPOLE! travel and tourism database. Since our 6 Domain Action Classification goal was to evaluate the SA and concept sequence classifiers and not segmentation, we created train- Identifying the domain action is a critical step in ing examples for each SDU in the database rather the analysis process for our interlingua-based than for each utterance. Table 4 contains statistics translation systems. One possible approach would regarding the contents of the corpus for our classi- be to manually develop grammars designed to fication tasks. Table 5 shows the frequency of the parse input utterances all the way to the domain most common domain action, speech act, and con- action level. However, while grammar-based pars- cept sequence in the corpus. These frequencies ing may provide very accurate analyses, it is gen- provide a baseline that would be achieved by a erally not feasible to develop a grammar that simple classifier that always returned the most completely covers a domain. This problem is exac- common class. erbated with spoken input, where disfluencies and deviations from the grammar are very common. English German Furthermore, a great deal of effort by human ex- SDUs 8289 8719 perts is generally required to develop a wide- Domain Actions 972 1001 coverage grammar. An alternative to writing full domain action Speech Acts 70 70 grammars is to train classifiers to identify the DA. Concept Sequences 615 638 Machine learning approaches allow the analyzer to Vocabulary Size 1946 2815 generalize beyond training data and tend to de- Table 4: Corpus Statistics. grade gracefully in the face of noisy input. Ma- chine learning methods may, however, be less English German accurate than grammars, especially on common in- DA (acknowledge) 19.2% 19.7% domain input, and may require a large amount of SA (give-information) 41.4% 40.7% training data in order to achieve adequate levels of Concept Sequence 38.9% 40.3% performance. In the hybrid analyzer described (No concepts) above, classifiers are used to identify the DA for Table 5: Most frequent DAs, SAs, and CSs. domain specific portions of utterances that are not covered by the cross-domain grammar. All of the results presented in this paper were We tested classifiers trained to classify com- produced using a 20-fold cross validation setup. plete DAs. We also split the DA classification task The corpus was randomly divided into 20 sets of into two subtasks: speech act classification and equal size. Each of the sets was held out as the test concept sequence classification. This simplifies the set for one fold with the remaining 19 sets used as task of each classifier, allows for the use of differ- training data. Within each language, the same ran- dom split was used for all of the classification ex- tures described above and produced the single best periments. Because the same split of the data was class as output. The SNNS classifier used a simple used for different classifiers, the results of two feed-forward network with 1 input unit for each classifiers on the same test set are directly compa- binary feature, 1 hidden layer containing 15 units, rable. Thus, we tested for significance using two- and 1 output unit for each speech act. The network tailed matched pair t-tests. was trained using backpropagation. The order of presentation of the training examples was random- 7.2 Machine Learning Approaches ized in each epoch, and the weights were updated We evaluated the performance of four different after each training example presentation. In order machine-learning approaches on the DA classifica- to simulate the binary features used by the other tion tasks: memory-based learning (k-Nearest- classifiers as closely as possible, the Rainbow clas- Neighbor), decision trees, neural networks, and sifier used a simple unigram model whose vocabu- naïve Bayes n-gram classifiers. We selected these lary was the set of labels included in the binary approaches because they vary substantially in the feature set. The setup for the DA classification ex- their representations of the training data and their periment was identical except that the neural net- methods for selecting the best class. work had 50 hidden units. Our purpose was not to implement each ap- The setup of the classifiers for the concept se- proach from scratch but to test the approach for our quence classification experiment was very similar. particular task. Thus, we chose to use existing The TiMBL and C4.5 classifiers were set up ex- software for each approach “off the shelf.” The actly as in the DA and SA experiments with one ease of acquiring and setting up the software influ- extra feature whose value was the speech act. The enced our choice. Furthermore, the ease of incor- SNNS concept sequence classifier used a similar porating the software into our online translation network with 50 hidden units. The SA feature was system was also a factor. represented as a set of binary input units. The Our memory-based classifiers were imple- Rainbow classifier was set up exactly as in the DA mented using TiMBL (Daelemans et al., 2002). We and SA experiments. The SA feature was not in- used C4.5 (Quinlan, 1993) for our decision tree cluded. classifiers. Our neural network classifiers were As mentioned above, both experiments used a implemented using SNNS (Zell et al., 1998). We 20-fold cross-validation setup. In each fold, the used Rainbow (McCallum, 1996) for our naïve TiMBL, C4.5, and Rainbow classifiers were sim- Bayes n-gram classifiers. ply trained on 19 subsets of the data and tested on the remaining set. The SNNS classifiers required a 8 Experiments more complex setup to determine the number of epochs to train the neural network for each test set. In our first experiment, we compared the perform- Within each fold, a cross-validation setup was used ance of the four machine learning approaches. to determine the number of training epochs. Each Each SDU was parsed using the argument and of the 19 training subsets for a fold was used as a pseudo-argument grammars described above. The validation set. The network was trained on the re- feature set for the DA and SA classifiers consisted maining 18 subsets until the accuracy on the vali- of binary features indicating the presence or ab- dation set did not improve for 50 consecutive sence of labels from the grammars in the parse for- epochs. The network was then trained on all 19 est for the SDU. The feature set included 212 training subsets for the average number of epochs features for English and 259 features for German. from the validation sets. This process was used for The concept sequence classifiers used the same all 20-folds in the SA classification experiment. feature set with the addition of the speech act. For the DA and concept sequence experiments, this In the SA classification experiment, the TiMBL process ran for approximately 1.5 days for each classifier used the IB1 (k-NN) algorithm with 1 fold. Thus, this process was run for the first two neighbor and gain ratio feature weighting. The folds, and the average number of epochs from C4.5 classifier required at least one instance per those folds was used for training. branch and used node post-pruning. Both the TiMBL and C4.5 classifiers used the binary fea- English German English German TiMBL 49.69% 46.51% Domain Action 48.59% 48.09% C4.5 48.90% 46.58% Speech Act 79.00% 77.46% SNNS 49.39% 46.21% Concept Sequence 56.87% 57.77% Rainbow 39.74% 38.32% Table 9: Rainbow accuracy with word bigrams. Table 6: Domain Action classifier accuracy. Table 9 shows the average accuracy of the English German Rainbow word bigram classifiers using the same TiMBL 69.82% 67.57% 20-fold cross-validation setup as in the previous C4.5 70.41% 67.90% experiments. As we expected, using word bigrams rather than parse label unigrams improved the per- SNNS 71.52% 67.61% formance of the Rainbow classifiers. For German 46.00% Rainbow 51.39% DA classification, the word bigram classifier was Table 7: Speech Act classifier accuracy. significantly better than all of the previous German DA classifiers (at least p=0.005). Furthermore, the English German Rainbow word bigram SA classifiers for both lan- TiMBL 69.59% 67.08% guages outperformed all of the SA classifiers that C4.5 68.47% 66.45% used only the parse labels. SNNS 71.35% 68.67% Although the argument parse labels provide an Rainbow 51.64% 51.50% abstraction of the present in an SDU, the Table 8: Concept Sequence classifier accuracy. words themselves also clearly provided useful in- formation for classification, at least for the SA Table 6, Table 7, and Table 8 show the average task. Thus, we conducted additional experiments to accuracy of each learning approach on the 20-fold examine whether combining parse and word in- cross validation experiments for domain action, formation could further improve performance. speech act, and concept classification respectively. We chose to incorporate word information into For DA classification, there were no significant the TiMBL classifiers used in the first experiment. differences between the TiMBL, C4.5, and SNNS Although the SNNS SA classifier performed sig- classifiers for English or German. In the SA ex- nificantly better than the TiMBL SA classifier for periment, the difference between the TiMBL and English, there was no significant difference for SA C4.5 classifiers for English was not significant. classification in German. Furthermore, because of The SNNS classifier was significantly better than the complexity and time required for training with both TiMBL and C4.5 (at least p=0.0001). For SNNS, we preferred working with TiMBL. German SA classification, there were no signifi- We tested two approaches to adding word in- cant differences between the TiMBL, C4.5, and formation to the TiMBL classifier. In both ap- SNNS classifiers. For concept sequence classifica- proaches, the word-based information for each fold tion, SNNS was significantly better than TiMBL was computed only based on the data in the train- and C4.5 (at least p=0.0001) for both English and ing set. In our first approach, we added binary fea- German. For English only, TiMBL was signifi- tures for the 250 words that had the highest mutual cantly better than C4.5 (p=0.005). information with the class. Each feature indicated For both languages, the Rainbow classifier per- the presence or absence of the word in the SDU. In formed much worse than the other classifiers. this condition, we used the TiMBL classifier with However, the unigram model over arguments did gain ratio feature weighting, 3 neighbors, and un- not exploit the strengths of the n-gram classifica- weighted voting. The second approach we tested tion approach. Thus, we ran another experiment in combined the Rainbow word bigram classifier with which the Rainbow classifier was trained on sim- the TiMBL classifier. We added one input feature ple word bigrams. No stemming or stop words for each possible speech act to the TiMBL classi- were used in building the bigram models. fier. The value of each SA feature was the prob- ability of the speech act computed by the Rainbow word bigram classifier. In this condition, we used the TiMBL classifier with gain ratio feature information with the class. We used a TiMBL clas- weighting, 11 neighbors, and inverse linear dis- sifier with gain ratio feature weighting and one tance weighted voting. neighbor. The improvement in accuracy for both languages was highly significant. English German TiMBL + words 78.59% 75.98% English German TiMBL + Rainbow 81.25% 78.93% TiMBL SA 49.63% 46.50% Table 10: Word+Parse SA classifier accuracy. + TiMBL CS TiMBL+Rainbow SA 57.74% 53.93% Table 10 shows the average accuracy of the SA + TiMBL CS classifiers that combined parse and word informa- Table 12: DA accuracy of SA+CS classifiers. tion using the same 20-fold cross-validation setup as the previous experiments. Although adding bi- Finally, Table 12 shows the results from two nary features for individual words improved per- tests to compare the performance of combining the formance over the classifiers with no word best output of the SA and concept sequence classi- information, it did not allow the combined classifi- fiers with the performance of the complete DA ers to outperform the Rainbow word bigram classi- classifiers. In the first test, we combined the output fiers. However, for both languages, adding the from the TiMBL SA and CS classifiers shown in probabilities computed by the Rainbow bigram Table 7 and Table 8. The performance of the com- model resulted in a SA classifier that outperformed bined SA+CS classifiers was almost identical to all previous classifiers. The improvement in accu- that of the TiMBL DA classifiers shown in Table racy was highly significant for both languages. 6. In the second test, we combined our best SA We conducted a similar experiment for combin- classifier (TiMBL+Rainbow, shown in Table 10) ing parse and word information in the concept se- with the TiMBL CS classifier. In this case, we had quence classifiers. The first condition was mixed results. The performance of the combined analogous to the first condition in the combined classifiers was better than our best DA classifier SA classification experiment. The second condi- for English and worse for German. tion was slightly different. A concept sequence can be broken down into a set of individual concepts. 9 Discussion The set of individual concepts is much smaller than the set of concept sequences (110 for English and One of our main goals was to determine the feasi- 111 for German). Thus, we used a Rainbow word bility of automatically classifying domain actions. bigram classifier to compute the probability of As the data in Table 4 show, DA classification is a each individual concept rather than the complete challenging problem with approximately 1000 concept sequence. The probabilities for the indi- classes. Even when the task is divided into vidual concepts were added to the parse label fea- subproblems of identifying the SA and concept tures for the combined classifier. In both sequence, the subtasks remain difficult. The diffi- conditions, the performance of the combined clas- culty is compounded by relatively sparse training sifiers was roughly the same as the classifiers that data with unevenly distributed classes. Although used only parse labels as features. the most common classes in our training corpus had over 1000 training examples, many of the English German classes had only 1 or 2 examples. TiMBL + words 56.48% 54.98% Despite these difficulties, our results indicate Table 11: Word+Parse DA classifier accuracy. that domain action classification is feasible. For SA classification in particular we were able to Table 11 shows the average accuracy of DA achieve very strong performance. Although per- classifiers for English and German using a setup formance on concept sequence and DA classifica- similar to the first approach in the combined SA tion is not as high, it is still quite strong, especially experiment. In this experiment, we added binary given that there are an order of magnitude more features for the 250 words that the highest mutual classes than in SA classification. Based on our ex- periments, it appears that all of the learning ap- proaches we tested were able to cope with data The machine learning approach should also be sparseness at the level found in our data, with the able to easily accommodate both continuous and possible exception of the naïve Bayes n-gram ap- discrete features from a variety of sources. Possible proach (Rainbow) for the concept sequence task. sources for features include words and/or phrases One additional point worth noting is that there in an utterance, the argument parse, the interlingua is evidence that domain action classification could representation of the arguments, and properties of be performed reasonably well using only word- the dialogue (e.g. speaker tag). The classifier based information. Although our best-performing should be able to easily combine features from any classifiers combined word and argument parse in- or all of these sources. formation, the naïve Bayes word bigram classifier Another desirable attribute for the machine (Rainbow) performed very well on the SA classifi- learning approach is the ability to produce a ranked cation task. With additional data, the performance list of possible classes. Our interlingua specifica- of the concept sequence and DA word bigram clas- tion defines how speech acts and concepts are al- sifiers could be expected to improve. Cattoni et al. lowed to combine as well as how arguments are (2001) also apply statistical language models to licensed by the domain action. These constraints DA classification. A word bigram model is trained can be used to select an alternative DA if the best for each DA, and the DA with the highest likeli- DA violates the specification. hood is assigned to each SDU. Arguments are Based on all of these considerations, the identified using recursive transition networks, and TiMBL+Rainbow classifier, which combines parse interlingua specification constraints are used to label features with word bigram probabilities, find the most likely valid interlingua representa- seems like an excellent choice for speech act clas- tion. Although it is clear that argument information sification. It was the most accurate classifier that is useful for the task, it appears that words alone we tested. Furthermore, the main TiMBL classifier can be used to achieve reasonable performance. meets all of the requirements discussed above ex- Another goal of our experiments was to help in cept the ability to produce a complete ranked list of the selection of a machine learning approach to be the classes for each instance. However, such a list used in our hybrid analyzer. Certainly one of the could be produced as a backup from the Rainbow most important considerations is how well the probability features. Adding new features to the learning approach performs the task. For SA classi- combined classifier would also be very easy be- fication, the combination of parse features and cause TiMBL was the primary classifier in the word bigram probabilities clearly gave the best combination. Finally, since both TiMBL and Rain- performance. For concept sequence classification, bow provide an online server mode for classifying no learning approach clearly outperformed any single instances, incorporating the combined clas- other (with the exception that the naïve Bayes n- sifier into an online translation system would not gram approach performed worse than other ap- be difficult. Since there were no significant differ- proaches). However, the performance of the classi- ences in the performance of most of the concept fiers is not the only consideration to be made in sequence classifiers, this combined approach is selecting the classifier for our hybrid analyzer. probably also a good option for that task. Several additional factors are also important in selecting the particular machine learning approach 10 Conclusion to be used. One important attribute of the learning approach is the speed of both classification and We have described a representation of speaker training. Since the classifiers are part of a transla- intention that includes domain independent speech tion system designed for use between two humans acts as well as domain dependent domain actions. to facilitate (near) real-time , the We have shown that domain actions are a useful DA classifiers must classify individual utterances level of abstraction for machine translation of task- online very quickly. Furthermore, since humans oriented dialogue, and that, in spite of their domain must write and test the argument grammars, specificity, they are scalable to larger domains and training and batch classification should be fast so portable to new domains. that the grammar writers can update the grammars, We have also presented classifiers for domain retrain the classifiers, and test efficiently. actions that have been comparatively tested and used successfully in the NESPOLE speech-to- ing Phrase-Level Grammars and Trainable Classifi- speech translation system. We experimentally ers. In Workshop on Algorithms for Speech-to- compared the effectiveness of several machine- Speech Machine Translation at ACL-02. Philadel- learning approaches for classification of domain phia, PA. actions, speech acts, and concept sequences on two Lavie, A., F. Metze, F. Pianesi, et al. 2002. Enhancing input languages. Despite the difficulty of the clas- the Usability and Performance of NESPOLE! – a sification tasks due to a large number of classes Real-World Speech-to-Speech Translation System. In and relatively sparse data, the classifiers exhibited Proceedings of HLT-2002. San Diego, CA. strong performance on all tasks. We also demon- Levin, L., D. Gates, A. Lavie, A. Waibel. 1998. An In- strated how the combination of two learning ap- terlingua Based on Domain Actions for Machine proaches could be used to improve performance Translation of Task-Oriented Dialogues. In Proceed- and overcome the weaknesses of the individual ings of ICSLP 98, Vol. 4, pages 1155-1158, Sydney, approaches. Australia. Levin, L., D. Gates, D. Wallace, K. Peterson, A. Lavie Acknowledgements: NESPOLE was funded F. Pianesi, E. Pianta, R. Cattoni, N. Mana. 2002. Bal- by NSF (Grant number 9982227) and the EU. The ancing Expressiveness and Simplicity in an Interlin- NESPOLE partners are ITC-irst, Universite Joseph gua for Task Based Dialogue. In Proceedings of Fourrier, Universitat Karlsruhe, APT Trentino Workshop on Spoken Language Translation. ACL- travel board, and AETHRA telecommunications. 02, Philadelphia. We would like to acknowledge the contribution of McCallum, A. K. 1996. Bow: A toolkit for statistical the following people in particular: Fabio Pianesi, language modeling, text retrieval, classification and Emanuele Pianta, Nadia Mana, and Herve Blan- clustering. http://www.cs.cmu.edu/~mccallum/bow. chon. Qu, Y., B. DiEugenio, A. Lavie, L. Levin and C.P. Rose. 1997. Minimizing Cumulative Error in Dis- References course Context. In Dialogue Processing in Spoken Language Systems: Revised Papers from ECAI-96 Cattoni, R., M. Federico, and A. Lavie. 2001. Robust Workshop, E. Maier, M. Mast and S. LuperFoy Analysis of Spoken Input Combining Statistical and (eds.), LNCS series, Springer Verlag. Knowledge-Based Information Sources. In Proceed- ings of the IEEE ASRU Workshop, Trento, Italy. Qu, Y., C. P. Rose, and B. DiEugenio. 1996. Using Dis- course Predictions for Ambiguity Resolution. In Pro- Daelemans, W., J. Zavrel, K. van der Sloot, and A. van ceedings of COLING-1996. den Bosch. 2002. TiMBL: Tilburg Memory Based Learner, version 4.3, Reference Guide. ILK Techni- Quinlan, J. R. 1993. C4.5: Programs for Machine Learn- cal Report 02-10. Available from ing. San Mateo: Morgan Kaufmann. http://ilk.kub.nl/downloads/pub/papers/ilk0210.ps.gz. Reithinger, N., R. Engel, M. Kipp, M. Klesen. 1996. Fillmore, C.J., Kay, P. and O'Connor, M.C. 1988. Regu- Predicting Dialogue Acts for a Speech-To-Speech larity and Idiomaticity in Grammatical Constructions. Translation System. DFKI GmbH Saarbruecken. Language, 64(3), 501-538. Verbmobil-Report 151. http://verbmobil.dfki.de/cgi- bin/verbmobil/htbin/doc-access.cgi Gavaldà, M. 2000. SOUP: A Parser for Real-World Spontaneous Speech. In Proceedings of IWPT-2000, Stolcke, A., K. Ries, N. Coccaro, E. Shriberg, R. Bates, Trento, Italy. D. Jurafsky, P. Taylor, R. Martin, M. Meteer, and C. Van Ess-Dykema. 2000. Dialogue Act Modeling for Goldberg, Adele E. 1995. Constructions: A Construc- Automatic Tagging and Recognition of Conversa- tion Grammar Approach to Argument Structure. Chi- tional Speech. Computational 26:3, 339- cago University Press. 371. Langley, C. and A. Lavie. 2003. Parsing Domain Ac- Zell, A., G. Mamier, M. Vogt, et al. 1998. SNNS: tions with Phrase-Level Grammars and Memory- Stuttgart Neural Network Simulator User Manual, Based Learners. To appear in Proceedings of IWPT- Version 4.2. 2003. Nancy, France. Langley, C., A. Lavie, L. Levin, D. Wallace, D. Gates, and K. Peterson. 2002. Spoken Language Parsing Us-