US010706841B2 (12 ) United States Patent ( 10 ) Patent No.: US 10,706,841 B2 Gruber et al. (45 ) Date of Patent : * Jul. 7 , 2020 (54 ) TASK FLOW IDENTIFICATION BASED ON (52 ) U.S. CI. USER INTENT CPC GIOL 15/1815 (2013.01 ) ; B60K 35/00 ( 2013.01 ) ; G06F 3/167 (2013.01 ) ; (71 ) Applicant: Apple Inc., Cupertino , CA (US ) ( Continued ) (58 ) Field of Classification Search (72 ) Inventors : Thomas Robert Gruber , Emerald CPC GIOL 15/22 ; G10L 15/28 ; G1OL 15/285 ; Hills , CA (US ) ; Adam John Cheyer , GIOL 15/30 ; G1OL 15/32 ; G1OL 15/34 Oakland, CA ( US ) ; Dag Kittlaus, See application file for complete search history . Cupertino , CA ( US ) ; Didier Rene Guzzoni, Mont sur- Rolle (CH ) ; (56 ) References Cited Christopher Dean Brigham , San Jose, CA (US ) ; Richard Donald Giuli, U.S. PATENT DOCUMENTS Arroyo Grande, CA ( US ) ; Marcello 1,559,320 A 10/1925 Hirsh Bastea -Forte , Cupertino , CA (US ) ; 2,180,522 A 11/1939 Henne Harry Joseph Saddler , Berkeley , CA ( Continued ) (US ) FOREIGN PATENT DOCUMENTS ( 73 ) Assignee : Apple Inc., Cupertino , CA (US ) CA 2694314 A1 8/2010 ( * ) Notice: Subject to any disclaimer , the term of this CA 2792412 A1 7/2011 patent is extended or adjusted under 35 ( Continued ) U.S.C. 154 ( b ) by 402 days. OTHER PUBLICATIONS This patent is subject to a terminal dis claimer . Advisory Action received for U.S. Appl .No. 13 /913,336 , dated Feb. 20 , 2018 , 3 pages . ( 21 ) Appl. No .: 15 /394,162 (Continued ) ( 22 ) Filed : Dec. 29 , 2016 Primary Examiner Michael N Opsasnick ( 74 ) Attorney , Agent, or Firm - Dentons US LLP (65 ) Prior Publication Data US 2017/0178626 A1 Jun . 22 , 2017 (57 ) ABSTRACT The intelligent automated assistant system engages with the user in an integrated , conversational manner using natural Related U.S. Application Data language dialog , and invokes external services when appro priate to obtain information or perform various actions . The (63 ) Continuation of application No. 13 / 492,809 , filed on system can be implemented using any of a number of Jun . 9 , 2012 , now Pat. No. 9,548,050 , which is a different platforms, such as the web , email, smartphone , and ( Continued ) the like , or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks, ( 51 ) Int . Ci. GIOL 15/22 ( 2006.01 ) and employs additional functionally powered by external GIOL 15/18 ( 2013.01 ) services with which the system can interact . (Continued ) 60 Claims, 47 Drawing Sheets 4.1 Onion Dinnye Die Kenici , WONOGI: VS . TO Fow's 1975 Pia kui Pirata US 10,706,841 B2 Page 2 Related U.S. Application Data 4,513,435 A 4/1985 Sakoe et al. 4,577,343 A 3/1986 Oura continuation of application No. 12 / 987,982 , filed on 4,586,158 A 4/1986 Brandle Jan. 10 , 2011, now Pat. No. 9,318,108 . 4,587,670 A 5/1986 Levinson et al. 4,589,022 A 5/1986 Prince et al. (60 ) Provisional application No. 61/ 295,774 , filed on Jan. 4,611,346 A 9/1986 Bednar et al. 4,615,081 A 10/1986 Lindahl 18 , 2010 . 4,618,984 A 10/1986 Das et al . 4,642,790 A 2/1987 Minshull et al. ( 51 ) Int . Ci. 4,653,021 A 3/1987 Takagi G06F 16/332 (2019.01 ) 4,654,875 A 3/1987 Srihari et al . GOOF 16/33 ( 2019.01 ) 4,655,233 A 4/1987 Laughlin G06F 16/9537 ( 2019.01 ) 4,658,425 A 4/1987 Julstrom 4,670,848 A 6/1987 Schramm GO6F 1/30 ( 2020.01 ) 4,677,570 A 6/1987 Taki G06F 40/40 (2020.01 ) 4,680,429 A 7/1987 Murdock et al. G06F 40/279 ( 2020.01) 4,680,805 A 7/1987 Scott G06F 3/16 ( 2006.01 ) 4,686,522 A 8/1987 Hernandez et al . GO6F 9/54 4,688,195 A 8/1987 Thompson et al. ( 2006.01 ) 4,692,941 A 9/1987 Jacks et al . GIOL 15/26 ( 2006.01 ) 4,698,625 A 10/1987 McCaskill et al . GIOL 21/06 ( 2013.01 ) 4,709,390 A 11/1987 Atal et al. BOOK 35/00 ( 2006.01) 4,713,775 A 12/1987 Scott et al . GIOL 13/02 ( 2013.01 ) 4,718,094 A 1/1988 Bahl et al. GIOL 13/08 4,724,542 A 2/1988 Williford ( 2013.01 ) 4,726,065 A 2/1988 Froessl GIOL 15/183 ( 2013.01 ) 4,727,354 A 2/1988 Lindsay H04M 1/60 ( 2006.01 ) RE32,632 E 3/1988 William H04M 1/725 ( 2006.01 ) 4,736,296 A 4/1988 Katayama et al . GIOL 13/00 4,750,122 A 6/1988 Kaji et al. ( 2006.01 ) 4,754,489 A 6/1988 Bokser (52 ) U.S. CI. 4,755,811 A 7/1988 Slavin et al . CPC G06F 9/54 (2013.01 ) ; G06F 16/3329 4,759,070 A 7/1988 Voroba et al . ( 2019.01 ) ; G06F 16/3344 ( 2019.01 ) ; G06F 4,776,016 A 10/1988 Hansen 4,783,804 A 11/1988 Juang et al . 16/9537 ( 2019.01) ; G06F 40/279 ( 2020.01) ; 4,783,807 A 11/1988 Marley G06F 40/30 (2020.01 ) ; G06F 40/40 4,785,413 A 11/1988 Atsumi ( 2020.01 ) ; GIOL 13/02 (2013.01 ) ; GIOL 13/08 4,790,028 A 12/1988 Ramage (2013.01 ) ; GIOL 15/183 (2013.01 ) ; GIOL 4,797,930 A 1/1989 Goudie 15/22 ( 2013.01 ) ; GIOL 15/26 (2013.01 ) ; GIOL 4,802,223 A 1/1989 Lin et al. 4,803,729 A 2/1989 Baker 15/265 (2013.01 ) ; GIOL 21/06 ( 2013.01) ; 4,807,752 A 2/1989 Chodorow H04M 1/6091 (2013.01 ) ; H04M 1/72547 4,811,243 A 3/1989 Racine ( 2013.01 ) ; H04M 1/72563 (2013.01 ) ; H04M 4,813,074 A 3/1989 Marcus 1/72597 ( 2013.01) ; GIOL 13/00 ( 2013.01) ; 4,819,271 A 4/1989 Bahl et al. GIOL 15/1822 ( 2013.01 ) 4,827,518 A 5/1989 Feustel et al. 4,827,520 A 5/1989 Zeinstra 4,829,576 A 5/1989 Porter ( 56 ) References Cited 4,829,583 A 5/1989 Monroe et al . 4,831,551 A 5/1989 Schalk et al . U.S. PATENT DOCUMENTS 4,833,712 A 5/1989 Bahl et al . 4,833,718 A 5/1989 Sprague 2,495,222 A 1/1950 Bierig 4,837,798 A 6/1989 Cohen et al . 3,704,345 A 11/1972 Coker et al . 4,837,831 A 6/1989 Gillick et al . 3,710,321 A 1/1973 Rubenstein 4,839,853 A 6/1989 Deerwester et al . 3,787,542 A 1/1974 Gallagher et al. 4,852,168 A 7/1989 Sprague 3,828,132 A 8/1974 Flanagan et al. 4,862,504 A 8/1989 Nomura 3,979,557 A 9/1976 Schulman et al . 4,875,187 A 10/1989 Smith 4,013,085 A 3/1977 Wright 4,878,230 A 10/1989 Murakami et al . 4,081,631 A 3/1978 Feder 4,887,212 A 12/1989 Zamora et al . 4,090,216 A 5/1978 Constable 4,896,359 A 1/1990 Yamamoto et al . 4,107,784 A 8/1978 Van Bemmelen 4,903,305 A 2/1990 Gillick et al . 4,108,211 A 8/1978 Tanaka 4,905,163 A 2/1990 Garber et al. 4,159,536 A 6/1979 Kehoe et al. 4,908,867 A 3/1990 Silverman 4,181,821 A 1/1980 Pirz et al . 4,914,586 A 4/1990 Swinehart et al. 4,204,089 A 5/1980 Key et al. 4,914,590 A 4/1990 Loatman et al. 4,241,286 A 12/1980 Gordon 4/1990 Cole et al. 4,253,477 A 3/1981 Eichman 4,914,704 A 4,278,838 A 7/1981 Antonov 4,918,723 A 4/1990 Iggulden et al. 4,282,405 A 8/1981 Taguchi 4,926,491 A 5/1990 Maeda et al . 4,310,721 A 1/1982 Manley et al . 4,928,307 A 5/1990 Lynn 4,332,464 A 6/1982 Bartulis et al . 4,931,783 A 6/1990 Atkinson 4,348,553 A 9/1982 Baker et al. 4,935,954 A 6/1990 Thompson et al. 4,384,169 A 5/1983 Mozer et al. 4,939,639 A 7/1990 Lee et al. 4,386,345 A 5/1983 Narveson et al . 4,941,488 A 7/1990 Marxer et al . 4,433,377 A 2/1984 Eustis et al . 4,944,013 A 7/1990 Gouvianakis et al. 4,451,849 A 5/1984 Fuhrer 4,945,504 A 7/1990 Nakama et al . 4,485,439 A 11/1984 Rothstein 4,953,106 A 8/1990 Gansner et al. 4,495,644 A 1/1985 Parks et al . 4,955,047 A 9/1990 Morganstein et al. 4,513,379 A 4/1985 Wilson et al . 4,965,763 A 10/1990 Zamora US 10,706,841 B2 Page 3 ( 56 ) References Cited 5,179,652 A 1/1993 Rozmanith et al. 5,194,950 A 3/1993 Murakami et al. U.S. PATENT DOCUMENTS 5,195,034 A 3/1993 Garneau et al. 5,195,167 A 3/1993 Bahl et al. 4,972,462 A 11/1990 Shibata 5,197,005 A 3/1993 Shwartz et al.