(12) United States Patent (10) Patent No.: US 9,318,108 B2 Gruber Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USOO931 8108B2 (12) United States Patent (10) Patent No.: US 9,318,108 B2 Gruber et al. (45) Date of Patent: Apr. 19, 2016 (54) INTELLIGENT AUTOMATED ASSISTANT G06F 17/30867; G06F 17/3087; G06F 17/30879; G06F 3/038; G10L 15/181; G 10L (75) Inventors: Thomas Robert Gruber, Emerald Hills, 15/19, G10L 15/30; G 10L 15/18: G 10L CA (US); Adam John Cheyer, Oakland, 15/1822; G 10L 15/20: G 10L 15/24: G10L CA (US); Dag Kittlaus, San Jose, CA 15/26 (US); Didier Rene Guzzoni, USPC .................... 704/270. 1, 275,235, 9, 10, 270; Mont-sur-Rolle (CH); Christopher 706/11: 707/909.01, 999.002; 705/28; Dean Brigham, San Jose, CA (US); 709/224, 217: 715/234, 727, 764,728; Richard Donald Giuli, Arroyo Grande, 379/201.01, 230; 455/556.1 CA (US); Marcello Bastea-Forte, New See application file for complete search history. York, NY (US); Harry Joseph Saddler, Berkeley, CA (US) (56) References Cited U.S. PATENT DOCUMENTS (73) Assignee: Apple Inc., Cupertino, CA (US) 1,559,320 A 10, 1925 Hirsh (*) Notice: Subject to any disclaimer, the term of this 2,180,522 A 11/1939 Henne patent is extended or adjusted under 35 U.S.C. 154(b) by 441 days. (Continued) FOREIGN PATENT DOCUMENTS (21) Appl. No.: 12/987,982 CA 2666438 C 6, 2013 (22) Filed: Jan. 10, 2011 CH 681573 A5 4f1993 Prior Publication Data (Continued) (65) OTHER PUBLICATIONS US 2012/OO16678A1 Jan. 19, 2012 Australian Office Action dated Nov. 13, 2012 for Application No. Related U.S. Application Data 2011205426, 7 pages. (Continued) (60) Provisional application No. 61/295,774, filed on Jan. 18, 2010. Primary Examiner — Vijay B Chawan (74) Attorney, Agent, or Firm — Morrison & Foerster LLP (51) Int. C. GIOL 2L/00 (2013.01) (57) ABSTRACT GIOL 5/8 (2013.01) An intelligent automated assistant system engages with the (Continued) user in an integrated, conversational manner using natural language dialog, and invokes external services when appro (52) U.S. C. priate to obtain information or perform various actions. The CPC ............ G10L 15/1815 (2013.01); G06F 3/167 system can be implemented using any of a number of differ (2013.01); G06F 9/54 (2013.01); ent platforms, such as the web, email. Smartphone, and the (Continued) like, or any combination thereof. In one embodiment, the (58) Field of Classification Search system is based on sets of interrelated domains and tasks, and CPC ..................... H04M 1/72561; H04M 2250/74; employs additional functionally powered by external services H04M 3/4938; G06F 17/30731: G06F with which the system can interact. 17/30241; G06F 17/30256; G06F 17/30265; 48 Claims, 47 Drawing Sheets 102 intelligent Automated Assistant Active Input Short Term Long Tem Elicitation Personal Personal i094 emory 052 Memory iO54 Domain Modeis 1086 Qutput to User input User 004 Wocatulary 008 158 Ontology Dialog Flow 1050 Models Language 1087 Pattern Recognizers 1060 Other / Other Events'Facts Dialog Flow Actions 1008 010 Domain Processor Entity 1080 Databases Output 1072 Processor Services 1090 Orchestration Language 1082 Interpreter 1073 Services 1084 US 9,318.108 B2 Page 2 (51) Int. Cl. 4,776,016 A 10/1988 Hansen G06F 7/30 2006.O1 4,783,804 A 11/1988 Juang et al. GIOL I5/22 ( ) 4,783,807 A 1 1/1988 Marley (2006.01) 4,785,413. A 1 1/1988 Atsumi G06F 7/28 (2006.01) 4,790,028 A 12/1988 Ramage G06F 3/16 (2006.01) 3.3w W4- A 3. SudiIn et al. G06F 9/54 (2006.01) 4,803,729 A 2/1989 Baker GIOL I5/26 (2006.01) 4,807,752 A 2/1989 Chodorow GIOL 2/06 (2013.01) 4,811,243 A 3, 1989 Racine 4,813,074. A 3/1989 Marcus (52) U.S. Cl. 4,819,271. A 4, 1989 Bahl et al. CPC ........... G06F 17/28 (2013.01); G06F 17/3087 4,827,518 A 5/1989 Feustel et al. (2013.01); G 10L 15/22 (2013.01); G 10L 15/26 4,827,520 A 5/1989 Zeinstra (2013.01); G 10L 2 1/06 (2013.01) 4,829,576 A 5/1989 Porter 4,829,583 A 5/1989 Monroe et al. 4,831,551 A 5/1989 Schalk et al. (56) References Cited 4,833,712 A 5/1989 Bahl et al. 4,833,718 A 5/1989 Sprague U.S. PATENT DOCUMENTS 4,837.798. A 6/1989 Cohen et al. 4,837,831. A 6/1989 Gillicket al. 3,704,345 A 11/1972 Coker et al. 4,839,853. A 6/1989 Deerwester et al. 3,710,321 A 1/1973 Rubenstein 4,852,168 A 7, 1989 Sprague 3,979,557 A 9, 1976 Schulman et al. 4875.87 A 10, 1989 Smith 4,013,085 A 3/1977 Wright 4,878,230 A 10/1989 Murakami et al. 4,081,631 A 3/1978 Feder 4,887.212 A 12/1989 Zamora et al. 4,107,784.2.87 A 8/1978ERA, Van Bemmelen 4,903,3054,896,359 A 2/19901/1990 YamamotoGillicket al. et al. 4,108,211 A 8/1978 Tanaka 4.905,163 A 2/1990 Garber et al. 4,181,821.;: A $21/1980 Pirzbit et al. al. 4.914,5864,908,867 A 4/19903/1990 SwinehartSilverman et al. 4.204,089 A 5/1980 Key et al. 4.914,590 A 4/1990 Loatman et al. 4,241.286 A 12, 1980 Gordon 4,918,723. A 4/1990 Iggulden et al. 4,253,477 A 3/1981 Eichman 4,926,491. A 5/1990 Maeda et al. 4.278,838 A 7/1981 Antonov 4,928.307 A 5/1990 Lynn 4,282.405 A 8/1981 Taguchi 4,931,783. A 6/1990 Atkinson 4,310,721. A 1/1982 Manley et al. 4,935,954. A 6/1990 Thompson et al. 4,332,464 A 6/1982 Bartulis et al. 4,939,639 A 7/1990 Lee et al. 4,348,553 A 9, 1982 Baker et al. 4.941,488 A 7/1990 Marxer et all 4,384,169 A 5/1983 Mozer et al. 4,944,013- I k A 7, 1990 Gouvianakis. et al. 4,386,345 A 5, 1983 Narveson et al. 4,945,504. A 7/1990 Nakama et al. 4.433,377 A 2f1984 Eustis et al. 4,953,106 A 8, 1990 Gansner et al. 4,451,849. A 5/1984 Fuhrer 4.955,047 A 9/1990 Morganstein et al. 4,485.439 A 1 1/1984 Rothstein 4,965,763 A 10/1990 Zamora 4,495,644. A 1/1985 Parks et al. 4.972.462. A 11, 1990 Shibata 4.513,379 A 4, 1985 Wilson et al. 4,974,191- 4 A 11/1990 Amirghodsi et al. 4,555,775343. A 1 1/1985 PikeSeet al. 4,977.598.4,975,975 A 12/1990 FilipskiDoddington et al. 4,577.343 A 3/1986 Oura 4,980,916 A 12/1990 Zinser 4,586, 158. A 4, 1986 Brandle 4,985,924. A 1/1991 Matsuura 4,587,670 A 5, 1986 Levinson et al. 4992.972 A 2f1991 Brooks etal 4,589,022 A 5, 1986 Prince et al. 4,994,966 A 2f1991 Hutchins 4,615,081.4,611,346 A 10,9/1986 1986 LindahlBednar et al. 4,994,983- - I A 2/1991 Landell et al. 4,618,984 A 10, 1986 Das et al. 387, A. E. E. ea 4,653,021 A 3/1987 Takagi WW t 4,654,875. A 3/1987 Srihari et al. 3.999; A 23: sial 4,658.425. A 4, 1987 Julstrom 5.020.112 A 5/199 Chou 4,670,848. A 6/1987 Schramm 5.021971. A 6/1991 Lindsay 4,677,570 A 6/1987 Taki 5,022,081. A 6/1991 Hirose et al. 2.85 A 23, Milocket al. 5,027,1 10 A 6/1991 Chang et al. 4,686,522s sw A 8/1987 Hernandez et al. 5,027.4068578. A 6/199136 E.Roberts et al. 4,688,195 A 8, 1987 Thompson et al. 5029.3 A 7, 1991 Ozawa 4,692,941. A 9, 1987 Jacks et al. 5,031.217. A 7/1991 Nishimura 4,698.625. A 10/1987 McCaskill et al. 5,032.989 A 7/1991 Tornetta 4,709,390 A 1 1/1987 Atal et al. 5,033,087. A 7/1991 Bahletal 4,713,775 A 12/1987 Scott et al. 5,040.2is A 81991 Vitale etal 4,718,094. A 1/1988 Bahl et al. w w itale et al. 4724.543 A 2, 1988 Williford 5,046,099 A 9, 1991 Nishimura 4,726,065 A 2/1988 Froessl 5,047,614 A 9/1991 Bianco 4.727,354 A 2, 1988 Lindsay 5,047,617 A 9, 1991 Shepard et al. RE32,633 E 3, 1988 William 5,050,215. A 9/1991 Nishimura 4,736,296 A 4, 1988 Katayama et al. 5,053,758 A 10, 1991 Cornett et al. 4,750,122 A 6/1988 Kaji et al. 5,054,084 A 10/1991 Tanaka et al. 4,754,489 A 6/1988 Bokser 5,057,915 A 10/1991 Kohorn et al. 4,755,811 A 7/1988 Slavin et al. 5,067,158 A 1 1/1991 Arjmand US 9,318.108 B2 Page 3 (56) References Cited 5,282,265 1, 1994 Rohra Suda et al.