US009208788B2 (12) United States Patent (10) Patent No.: US 9,208,788 B2 Lerner (45) Date of Patent: Dec. 8, 2015 (54) METHOD AND APPARATUS FOR 2006. A R. 358, EchegyenOaCS . .3.56 RESENDING TO A QUERY AT A DIALOG 2003/0046083 A1* 3/2003 Devinney et al. TO4,273 2003/0055638 A1* 3, 2003 Burns et al. .... ... 704/231 2003/O120517 A1* 6/2003 Eida et al. ......................... 705/3 (75) Inventor: Solomon Z. Lerner, Sharon, MA (US) 2005, 01321.96 A1* 6, 2005 Dietl .............. T13, 176 2005/01932.06 A1* 9, 2005 Kunisa et al. ...... T13, 176 (73) Assignee: Nuance Communications, Inc., 38885 A. 3. SEsalev etet al. al ....... .70% Burlington, MA (US) 2008/0273744 A1* 1 1/2008 Kogure et al. ... ... 38.2/100 (*) Notice: Subject to any disclaimer, the term of this 2008/0276265 A1* 11/2008 TopchvOpcny et aal. .................. 725/22 patent is extended or adjusted under 35 (Continued) U.S.C. 154(b) by 290 days. OTHER PUBLICATIONS (21) Appl. No.: 13/560,217 U.S. Appl. No. 13/477,594, filed May 22, 2012 by William F. Ganong, III, et al. (22) Filed: Jul. 27, 2012 “Easter egg (media).”from Wikipedia, http://en.wikipedia.org/wiki/ Easter egg (media), 6 pages, retrieved on May 17, 2012. (65) Prior Publication Data “Fictitious entry.” from Wikipedia, http://en.wikipedia.org/wiki/Fic titious entries, 6 pages, retrieved on May 17, 2012. US 2014/OO32220 A1 Jan. 30, 2014 (Continued) (51) Int. Cl. GIOL 2L/00 (2013.01) Primary Examiner — Shaun Roberts GOL 25/00 (2013.01) (74) Attorney, Agent, or Firm — Hamilton, Brook, Smith & GIOL 7/24 (2013.01) Reynolds, P.C. GIOL 1.5/22 (2006.01) (52) U.S. Cl. (57) ABSTRACT CPC G10L 17/24 (2013.01); G 10L 15/22 (2013.01) A dialog system is accessed by a remote user and is typically (58) Field of Classification Search configured to receive a natural language query from the user CPC ............................ G10L 17/005; G06F 10/121 and return a natural language answer to the user. Dialog USPC ................. 704/231, 273, 275; 705/51, 54, 57 systems can be copied without authorization or can become See application file for complete search history. an out-of-date version. A dialog system with a signature, referred to herein as a “signed dialog system, can indicate (56) References Cited the signature without affecting usage by users who are unaware that the dialog system contains the signature. The U.S. PATENT DOCUMENTS signed dialog system can respond to input such that only the 5,933,498 A * 8/1999 Schnecket al. ................. 705/54 designer of the dialog system knows the signature is embed 6,233,347 B1 * 5/2001 Chen et al. ....... ... 38.2/100 ded in the dialog system. The response is a way to check the 6,442,283 B1* 8/2002 Tewfik et al. ... ... 38.2/100 Source or other characteristics of the dialog system. A 6,640,145 B2 * 10/2003 Hoffberg et al. ... 700.83 designer of signed dialog systems can prove whetheran unau 6,892, 175 B1* 5/2005 Cheng et al. ......... 704/205 7,403,616 B2 * 7/2008 Kageyama et al. ........... 380,201 thorized copy of the signed dialog system is used by a third 7,499,566 B2 * 3/2009 Rhoads ................ ... 38.2/100 party by using publically-available user interfaces. 7,606,768 B2 * 10/2009 Graubart et al. ..... ... TOS/51 7,694,138 B2 * 4/2010 O’Gorman et al. ........... T13,168 20 Claims, 15 Drawing Sheets 162 FIRSTNETWORKDEVICE e.g., ANINTERACTIVE WOCERESPONSE (VR) SERVER) SET OFSIGNED MODES SECONDAETWORK DEMCE SETOFUNSIGNED MODELS 164 f THIRD NETWORK DEWICE SEIFSISNED MODELS (DIFFERENT WERSIONS) w 82 165 FOURTHNETWORK DECE SET OFSIGNED MODELS (UNAUTHORIZED) 183 US 9,208,788 B2 Page 2 (56) References Cited OTHER PUBLICATIONS U.S. PATENT DOCUMENTS Rabiner, Lawrence R., "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition.” Proceedings of the 2011/O144985 A1 6, 2011 Krishnakumar et al. ..... TO4/230 30,300 loss, Ai i2013 ROWAS. 704,556 IEEE, Vol. 77, No. 2, pp. 257-286 (Feb. 1989). 2012/O123786 A1* 5, 2012 Valin et al. .................... 704/273 2013/0317817 A1 1 1/2013 Ganong, III, et al. * cited by examiner U.S. Patent Dec. 8, 2015 Sheet 1 of 15 US 9.208,788 B2 162 100 y FIRST NETWORKDEVICE (e.g., AN INTERACTIVE VOICE RESPONSE (IVR) SERVER) NETWORK SET OF SIGNE D MODELS 180 SECOND NETWORK DEVICE SET OF UNSIGNED MODELS 181 THIRD NETWORK DEVICE SET OF SIGNED MODELS (DIFFERENT VERSIONS) FOURTH NETWORK DEVICE SET OF SIGNED MODELS CLIENT (UNAUTHORIZED) DEVICE 155 FIG. I.A. U.S. Patent Dec. 8, 2015 Sheet 2 of 15 US 9.208,788 B2 r-------------------------------------- U.S. Patent Dec. 8, 2015 Sheet 3 of 15 US 9.208,788 B2 202 SEND TESTING 200 INTERACTION 204 RECEIVE RESPONSE TO TESTING INTERACTION 206 IS TESTING 216 INTERACTION UNSIGNED CONFIGURED TO CAUSE RESPONSE DO NOT A SIGNATURE OR AN FLAG MODEL UNSIGNED RESPONSE SIGNATURE 208 DOES RECEIVED RESPONSE FSML INCLUDE EXPECTED SIGNATURE AUTHORIZED IS FLAG MODEL RESPONSE FROM AS BEING AED UNAUTHORIZED 210 FLAG MODELAS 218 BEING UNAUTHORIZED RETURN MODEL 214 STATUS TO USER FIG. 2 U.S. Patent Dec. 8, 2015 Sheet 5 Of 15 US 9.208,788 B2 400 402 RECEIVE INPUT 404 IS INPUT WATERMARKED ? DD INPUT ORIGINATE FROM SET RESPONSE A PARTICULAR TO REJECTED CLIENT2 406 408 410 ACCEPT INPUT ATRECOGNIZER RETURN RESPONSE TO 412 INTERACTION UNIT FIG. 4 U.S. Patent Dec. 8, 2015 Sheet 6 of 15 US 9.208,788 B2 INTERACTION TO CAUSE 502 500 \ FIRST SIGNATURE 2 504 PASS FIRST SIGNATURE 506 INTERACTION TO CAUSE 506 SECOND SIGNATURE / INTERACTION PS SECOND SIGNATURE 5*/ SERVER WITH DEVICE INTERACTION TO CAUSE RECOGNIZER 160 FIG. 5A 140 THIRD SIGNATURE AND MODEL e - UNDER Test PASS THIRD SIGNATURE 510 506 INTERACTION TO CAUSE FOURTHSIGNATURE 512 PASS FOURTHSIGNATURE 514 506 V 516 INTERACTION TO CAUSE 502 520 \ FIRST SIGNATURE / 504 PASS FIRST SIGNATURE 506 INTERACTION TO CAUSE 506 SECOND SIGNATURE / RESPONSE WITHOUT INDICATION 522 INTERACTION 5. OF SECOND SIGNATURE SERVER WITH DEVICE 6 INTERACTION TO CAUSE 510 RECOGNIZER 1609 FIG 5B 140 THIRD SIGNATURE / AND MODEL UNDER TEST RESPONSE WITHOUT INDICATION FAIL OF THIRD SIGNATURE 526 INTERACTIONICAUSEFOURTHSIGNATURE 51 U.S. Patent Dec. 8, 2015 Sheet 7 Of 15 US 9.208,788 B2 SERVER WITH RECOGNIZER INTERACTION DEVICE (140) (160) AND MODELUNDER TEST 600 620 602 622 604 DATA FROMA USER 624 60 5 INTERACTION ACCESS SIGNED DATA MODELS WITH 606 120 RECEIVED INTERACTION GENERATE OR LOAD ANY OF EXPECTED RESPONSE RESPONSES 626 135-137 RECEIVE SET OF RESPONSES FROM SET DOES OF SIGNED MODELS RESPONSE INCLUDESIGNATURE NO CORRESPONDING TO THE EXPECTED RESPONSE PROCESS SET OF 608 RESPONSES AND APPLY SELECT MOST LIKELY "SIGNATURES RESPONSE APPLY "SIGNATURE NOT PRESENT IS PRESENT. FLAG FLAG 628 610 614 REPORT FLAG WAIT FOR NEXT INPUT THEN REPEAT (622) 612 630 U.S. Patent US 9.208,788 B2 U.S. Patent Dec. 8, 2015 Sheet 9 Of 15 US 9.208,788 B2 800 802 ? 804 RECEIVE INSTRUCTIONS FROM USER 806 808 INSTRUCTIONS TO GENERATE GENERATESIGNED UNSIGNED MODEL MODEL2 INSTRUCTIONS GENERATESIGNED TO INCLUDE MULTIPLE MODEL WITH ONE SIGNATURES IN SIGNED MODEL SIGNATURE 810 GENERATESIGNED MODEL 814 WITH MULTIPLESIGNATURES U.S. Patent Dec. 8, 2015 Sheet 10 of 15 US 9.208,788 B2 UNSIGNED PHYSICS MODEL FIG. 9A 910 RESPONSE OF UNSIGNED MODEL 906 908 (OPTIONAL) TARGET TRIGGERING PATH SIGNED PHYSICS MODEL FIG. 9B 922 RESPONSE OF SIGNED MODEL 924 908 (OPTIONAL) TARGET TRIGGERING PATH U.S. Patent Dec. 8, 2015 Sheet 11 of 15 US 9.208,788 B2 UNSIGNED PHYSICS MODEL FIG. 9C 910 RESPONSE 942 INPUT VECTOR 944 FORCE OF GRAVITY 946 SIGNED PHYSICS MODEL FIG. 9D 922 POST-SINGULARITY RESPONSE 968 SINGULARITY RESPONSE 964 962 INPUT VECTOR 944 966 PRE-SINGULARITY RESPONSE ------------o OF GRAVITY 946 U.S. Patent US 9.208,788 B2 || || || TVNOlld0~);TROITWIST;|| || || U.S. Patent Dec. 8, 2015 Sheet 15 Of 15 US 9.208,788 B2 1300 1302 RECEIVE QUERY FROM CLIENT DEVICE 1304 DOES QUERY MATCHA STEREOGRAPHIC PATTERN? RETRIEVE ARESPONSE CORRESPONDING WITH THE RETRIEVE 1306 PATTERN, THE RESPONSE A NORMAL 1312 REPRESENTING ASIGNATURE RESPONSE IF TRIGGERED BY THE QUERY ENCRYPT THE RESPONSE 1308 (OPTIONALLY) 1310 TRANSMIT THE RESPONSE FIG. I.3 US 9,208,788 B2 1. 2 METHOD AND APPARATUS FOR acoustic models, language models, and Vocabulary list mod RESPONDING TO A QUERY AT A DIALOG els can be signed. As language changes or speech-to-text SYSTEM recognition improves, the acoustic models, language models, vocabulary list models, and all other models associated with BACKGROUND OF THE INVENTION speech-to-text recognition are updated. Because models can be distributed about a network, it is useful to control versions Remote servers that include recognizers and models have and authorized distribution of the models. Verifying version become increasingly popular since the advent of cloud com control and authorized distribution may be handled in the puting. These recognizers perform calculations using the form of embedding certain words in a signed model Such that models for connected client devices, which increases utility 10 later updates and Verifications can be performed through stan of client devices with little computing cost because client dard Voice interaction to a server with a recognizer employing devices can access computing power of remote servers the model. The embedding of words or other signatures can be through computationally simple communications via com thought as a form of Steganography. munications networks. Steganography can be used to detect misappropriation of 15 proprietary information, Such as intellectual property.