(12) United States Patent (10) Patent No.: US 9,081,411 B2 Kalns Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US009081411 B2 (12) United States Patent (10) Patent No.: US 9,081,411 B2 Kalns et al. (45) Date of Patent: Jul. 14, 2015 (54) RAPID DEVELOPMENT OF VIRTUAL 8,316,343 B2 * 1 1/2012 Birze et al. .................... 717/1OO PERSONAL ASSISTANT APPLICATIONS 2003/0005412 A1 1/2003 Eanes ........................... 717/120 2003/0046201 A1 3/2003 Cheyer 2005/0097.525 A1* 5, 2005 Stone et al. ................... 717/136 (71) Applicant: SR International, Menlo Park, CA 2007/0022408 A1 1/2007 Brandt et al. ................. 717/136 (US) (Continued) (72) Inventors: Edgar T. Kalns, San Jose, CA (US); Dayne B. Freitag, La Mesa, CA (US); FOREIGN PATENT DOCUMENTS William S. Mark, San Mateo, CA (US); WO WO2O11028844 A2 3, 2011 Necip Fazil Ayan, Palo Alto, CA (US); Michael J. Wolverton, Mountain View, OTHER PUBLICATIONS CA (US); Thomas J. Lee, San Francisco, CA (US) Jari Porras, Reusable Bluetooth Networking Component for Symbian Operating System, 2002, pp. 22-38.* (73) Assignee: SR international Menlo Park, CA (Continued) (*) Notice: Subject to any disclaimer, the term of this Primary Examiner — Thuy Dao patent is extended or adjusted under 35 U.S.C. 154(b) by 132 days Assistant Examiner — Mongbao Nguyen M YW- y yS. (74) Attorney, Agent, or Firm — Barnes & Thornburg LLP (21) Appl. No.: 13/891,858 57 ABSTRACT (22) Filed: May 10, 2013 (57) A platform for developing a virtual personal assistant (65) Prior Publication Data (“VPA) application includes an ontology that defines a com US 2014/0337814 A1 Nov. 13, 2014 puterized structure for representing knowledge relating to • - s one or more domains. A domain may refer to a category of (51) Int. Cl. information and/or activities in relation to which the VPA G06F 9/44 (2006.01) application may engage in a conversational natural language G06F 9/45 (2006.01) dialog with a computing device user. Re-usable VPA compo (52) U.S. Cl. nents may be linked to or included in the ontology. An ontol CPC. G06F 8/00 (2013.01); G06F 8/20 (2013.01); ogy populating agent may at least partially automate the GO6F 8/34 (2013.01). GO6F 8/36 (2013,015 process of populating the ontology with domain-specific (58) Field of Classification search information. The re-usable VPA components may be linked CPC GO6F 8/20: GO6F 8/3O GO6F 8/34 with the domain-specific information through the ontology. A - - - - - - - - - - - - - - - s s GO6F 8/3 6 VPA application created with the platform may include See application file for complete search histo domain-adapted re-usable VPA components that may be pp p ry. called upon by an executable VPA engine to determine a (56) References Cited likely intended meaning of conversational natural language input of the user and/or initiate an appropriate system U.S. PATENT DOCUMENTS response to the input. 7,047,524 B1* 5/2006 Braddock, Jr. ................ 717/136 7,353,502 B2* 4/2008 Stewart et al. ................ 717/136 19 Claims, 11 Drawing Sheets LOCALM008L 8REATOR38 Script Max-INE LEAMM3 ROCES38 ge 3ARASLE XMOL883''S 12 ELEMEN: &LASSIFIR438 ReCR3 kycitor 41a. Nerk 42 is fast 3:0 US 9,081.411 B2 Page 2 (56) References Cited Nitz et al., U.S. Appl. No. 13/585,008, “Method, System, and Device for Inferring a Mobile User's Context and Protectively Providing U.S. PATENT DOCUMENTS Assistance”, Apr. 14, 2012. Wolverton et al., U.S. Appl. No. 13/678,209, “Vehicle Personal 2007/0094638 A1* 4/2007 DeAngelis et al. ........... 717/107 Assistant', Nov. 15, 2012. 2007/01OO790 A1 5/2007 Cheyer et al. Wolverton et al., U.S. Appl. No. 13/678,213, “Vehicle Personal 2007/0250405 A1 * 10, 2007 Ronen et al. .................... 705/27 Assistant', Nov. 15, 2012. 2008/0256505 A1 * 10, 2008 Chaar et al. ................... 717/100 Judith Aquino, "Siri-ous Influence: Enterprise virtual Assistants 2008/026.3506 A1* 10, 2008 Broadfoot et al. ............ T17,104 Have Arrived'. Speech Technology Magazine, Nov./Dec. 2012, pp. 2009, O164441 A1 6/2009 Cheyer 13-16. 2009/0187425 A1* 7/2009 Thompson ........................ 705/3 Freitag, Dayne, et al., “Finding the What and the How Toward Auto 2010/0287525 A1* 11/2010 Wagner ....... 717/100 mated Model Construction from Text”. Final Technical Report on 2011/0099536 A1* 4/2011 Coldicott et al. ............. 717/120 HNCSoftware's Project on Relationship Extraction and Link Induc 2012fOO16678 A1 1/2012 Gruber tion for Simulation and Modeling (REALISM), under IARPA's pro 2012/0072884 A1 3/2012 Balko et al. ................... T17,104 gram on Proactive Intelligence (PAINT); Jan. 2009. 2012/0131542 A1* 5/2012 DeAngelis et al. ........... T17,104 “Nuance's Nina is both more and less than Siri', available at http:// 2012/013.7271 A1* 5, 2012 Kemmler ...................... 717/120 www.meisel-on-mobile.com/2012/08/06/nuances-nina-is-both 2012fO233593 A1* 9, 2012 Sahoo et al. 717/120 more-and-less-than-Siri/, Aug. 6, 2012, 1 page. 2012fO244507 A1* 9, 2012 Tu et al. ........................ 434,362 "Angel Unveils a More Personalized Customer Experience with 2012fO245944 A1 9, 2012 Gruber et al. Introduction of Lexee from Angel Labs', available at http://www. 2013/0036398 A1* 2/2013 Majumdar et al. ............ 717/1OO angel.com/docs/press-releases Angel Unveils More Personal 2013,0152092 A1 6/2013 Yadgar ized Customer Experience Lexee Angel Labs 2012 08-13. pdf, accessed Feb. 12, 2015, 3 pages. OTHER PUBLICATIONS “Interactive product brochure :: Nina: The virtual Assistant for Mobile Customer Services Apps'. Nuance communications, Inc., Walt Scacchi, Process Models in Software Engineering, 2001, pp. accessed Feb. 12, 2015, 11 pages. 1-16. Harvest the Web, OutWit Technologies, available at http://www. Deng-Jyi Chen, Integration of Reusable Software Components and outwit.com/, accessed Feb. 12, 2015, 1 page. Frameworks Into a Visual Software Construction Approach, 2000, Blanko et al., “Open Information Extraction from the Web”, Depart pp. 863-880.* ment of Computer Science and Engineering, University of Washing Nitz et al., U.S. Appl. No. 13/585,003, “Method, System, and Device ton, Seattle, WA. Dec. 2008, 7 pages. for Inferring a Mobile User's Context and Protectively Providing Assistance”, Apr. 14, 2012. * cited by examiner U.S. Patent Jul. 14, 2015 Sheet 1 of 11 US 9,081.411 B2 COMPUTING SYSTEM 1 OO VIRTUAL PERSONAL ASSISTANT DEVELOPMENT PLATFORM 110 ONTOLOGY VISUALIZATION MODULE 120 NHERITANCE REASONING MODULE 122 SHAREABLE ONTOLOGY 1 12 RE-USABLE VPA COMPONENTS 114 DOMAN KB 116 ONTOLOGY POPULATING AGENT 118 FIG. 1 U.S. Patent Jul. 14, 2015 Sheet 2 of 11 US 9,081.411 B2 COMPUTING SYSTEM 200 VIRTUAL PERSONAL ASSISTANT 21 O MULT-MODAL. USER INTERFACE 212 3. VPA ENGINE 214 USER INTENT OUTPUT NTERPRETER REASNER GENERATOR 216 220 DOMAN-ADAPTED RE-USABLE VPA COMPONENTS 222 NL GRAMMARS 224 NTENTS 226 INTERPRETER TASK FLOWS OUTPUT NL RESPONSES 234 EMPLATES 236 ACOUSC LANGUAGE STATESTICAL MODELS 238 MODELS 240 MODELS 242 FIG. 2 U.S. Patent Jul. 14, 2015 Sheet 3 of 11 US 9,081.411 B2 ONTOLOGY POPULATING AGENT 118 SEMANTFER 31 O C d LOCAL MODEL CREATOR 316 LOCAL NFERENCE DOMAN KB MODEL 314 ENGINE 322 312 SHAREABLE ONTOLOGY 112 MACHINE LEARNING MODELS 318 WEB PAGE 32O FIG. 3 U.S. Patent Jul. 14, 2015 Sheet 4 of 11 US 9,081.411 B2 LOCAL MODEL 314 LOCAL MODEL CREATOR 316 NORMALZER 420 USER INPUT MACHINE RECORD USER LEARNING INTERFACE CLASSIFER 418 MODELS 318 422 SHAREABLE ONTOLOGY 112 ELEMENT CLASSIFIER 416 RECORD LOCATOR 414 WEB PAGE 32O FIG. 4 U.S. Patent Jul. 14, 2015 Sheet 5 of 11 US 9,081.411 B2 500 510 TRIGGER TO POPULATE NO A DOMAIN ONTOLOGY? YES 512 DENTIFY RECORD AND DATA VALUES ON SOURCE INTERNET WEB PAGE FOREACH DATA VALUE, DETERMINEA 514 VALUE TYPE BASED ON THE LOCAL MODEL MAP THE RECORD, DATA VALUE, AND 516 VALUE TYPE TO THE DOMAIN ONTOLOGY 518 ANOTHER RECORD ON THE WEB PAGET 520 CONTINUE ONTOLOGY POPULATING NO FIG. 5 U.S. Patent Jul. 14, 2015 Sheet 6 of 11 US 9,081.411 B2 600 HERARCHICAL DOMAN ONTOLOGY 610 ROOT-LEVE DOMAN 612 ROO-LEVEL VPA COMPONENTS 614 ROO-LEVEL DOMAN CONTENT 616 LEVEL 2, 1 DOMAIN 618 LEVEL 2, X DOMAN 626 NHERED WPA NHERIED WPA COMPONENS 62O COMPONENTS 628 LEVEL 2, 1 VPA LEVEL 2,XVPA COMPONENTS COMPONENTS 622 630 LEVEL, N, 1 DOMAIN 634 LEVEL N.2 DOMAN 642 LEVEL NY DOMAN 650 NHERITED WPA NHERED WPA NHERED WPA COMPONENIS 636 COMPONENTS 644 COMPONENTS 652 LEVEL, N, 1 VPA LEVEL N2 VPA LEVELN, Y VPA COMPONENS COMPONENTS COMPONENTS 638 646 654 LEVEL, N, 1 LEVELN,2 LEVEL N, Y CONTENT 640 CONTENT 648 CONTENT 656 FIG. 6 U.S. Patent Jul. 14, 2015 Sheet 7 of 11 US 9,081.411 B2 NL DIALOG 740 User: What are people saying about E-COMMERCE ONTOLOGY 710 <product>? NL GRAMMAR 732 PURCHASABLE WPA: There are <tota I am looking for TEM 712 reviews> reviews, with <product an average <avg rating> rating. - product - Color - description User: Tell me about the <rating type - price tings - quantity ratingS. - availability - Customer reviews NL GRAMMAR (34 - Competitor terms I am looking for - shipping terms (style=) formal (product) dresses in (color=) black APPAREL ELECTRONICS 726 714. - make - model - specifications PHONES 73O JEANS 716 WOMENS 718 MENS 720 NL DIALOG 742 User: What are people saying about (make=) BOOT CUT 722 SLM 724 Apple (product=) iphones? VPA: There are 55 reviews, with an average NL GRAMMAR 736 4-star rating. I am looking for (style=) User: Tell me about the boot-cut (product=) jeans in (ColorF) acid-washed negative ratings. indigo FIG. 7 U.S. Patent Jul. 14, 2015 Sheet 8 of 11 US 9,081.411 B2 ASK FLOW 814 Put in Shopping Cart E-COMMERCE ONTOLOGY 710 NL INTENT 810 PURCHASABLE Buy Product TEM 712 (product, quantity) - product - Color - description Offer Alternative - price - quantity - availability - Customer reviews - Competitor tems - Shipping terms APPAREL ELECTRONICS 714 726 - make - model - specifications JEANS 716 PHONES 730 WOMENS 718 MENS 720 TASK FLOW 816 BOOT CUT 722 SLIM 724 Add to shopping cart (HDTV, Samsung, 55 inch, 1) NL INTENT 812 Buy Product (jeans, boot-cut, red, 1) FIG.