|HAO WANATAIKITOLUULUUN US009966065B2 OTTI (12 ) United States Patent ( 10 ) Patent No. : US 9 ,966 , 065 B2 Gruber et al. (45 ) Date of Patent: May 8 , 2018 ( 54 ) MULTI- COMMAND SINGLE UTTERANCE ( 56 ) References Cited INPUT METHOD U . S . PATENT DOCUMENTS (71 ) Applicant : Apple Inc. , Cupertino , CA (US ) 1 ,559 , 320 A 10 /1925 Hirsh ( 72 ) Inventors : Thomas R . Gruber , Emerald Hills , CA 2 , 180 , 522 A 11/ 1939 Henne ( US ) ; Harry J . Saddler , Berkeley , CA (Continued ) (US ) ; Jerome Rene Bellegarda , Saratoga , CA (US ) ; Bryce H . Nyeggen , FOREIGN PATENT DOCUMENTS San Mateo , CA (US ) ; Alessandro CA 2666438 C 6 /2013 Sabatelli , San Francisco , CA (US ) CH 681573 A5 4 / 1993 ( 73 ) Assignee : Apple Inc ., Cupertino , CA (US ) (Continued ) ( * ) Notice : Subject to any disclaimer , the term of this OTHER PUBLICATIONS patent is extended or adjusted under 35 Gurevych et al ., “ Semantic Coherence Scoring using an Ontology ” , U . S . C . 154 ( b ) by 0 days . days . North American Chapter of the Association for Computational Linguistics Archive , Proceedings of the 2003 Conference of the (21 ) Appl. No. : 14 /724 ,623 North American Chapter of the Association for Computational Linguistics on Human Language Technology , May 27, 2003, 8 ( 22 ) Filed : May 28 , 2015 pages . (65 ) Prior Publication Data ( Continued ) US 2015 /0348551 A1 Dec . 3 , 2015 Primary Examiner — Anne L Thomas -Homescu ( 74 ) Attorney, Agent, or Firm — Dentons US LLP Related U . S . Application Data (57 ) ABSTRACT (60 ) Provisional application No . 62 /005 ,556 , filed on May Systems and processes are disclosed for handling a multi 30 , 2014 , provisional application No . 62 /129 ,851 , part voice command for a virtual assistant. Speech input can filed on Mar. 8 , 2015 . be received from a user that includes multiple actionable commands within a single utterance. A text string can be (51 ) Int . Ci. generated from the speech input using a speech transcription GIOL 15 / 18 ( 2013 . 01 ) process . The text string can be parsed into multiple candi G06F 17 / 27 ( 2006 .01 ) date substrings based on domain keywords, imperative ( Continued ) verbs , predetermined substring lengths, or the like . For each (52 ) U . S . CI. candidate substring , a probability can be determined indi CPC .. GIOL 15/ 1822 ( 2013 .01 ) ; G06F 17/ 2705 cating whether the candidate substring corresponds to an ?êbabe (2013 .01 ) ; GIOL 15 /1815 (2013 .01 ) ; actionable command . Such probabilities can be determined (Continued ) based on semantic coherence , similarity to user request (58 ) Field of Classification Search templates , querying services to determine manageability , or CPC . GIOL 15 /22 ; GIOL 15 / 265 ; G10L 15 / 063: the like. If the probabilities exceed a threshold , the user GIOL 17 / 005 ; G10L 17 / 187 ; intent of each substring can be determined , processes asso (Continued ) (Continued ) Process 800 Receive speech input from a user 802 Generate a text string based on the speech 804 input using a speech transcription process Parse the text string into at least a first candidate substring and a second candidate - 806 substring Determine a first intent associated with the first candidate substring and a second inlent L 812 Determine a first probability that the first associated with the second candidate candidate substring corresponds to a first substring actionable command and a second 808 probability that the second candidate wwwwwwwwwwwww substring corresponds to a second Execute a first process associated with the actionable command first intent and a second process associated 814 with the second inient 810 no Probabilities Provide to the user an acknowledgment exceed Yes associated with the first intent and the 816 threshold ? second intent US 9 , 966 ,065 B2 Page 2 ciated with the user intents can be executed , and an acknowl 4 ,692 , 941 A 9 / 1987 Jacks et al. 4 ,698 ,625 A 10 / 1987 McCaskill et al . edgment can be provided to the user. 4 , 709 , 390 A 11/ 1987 Atal et al . 4 ,713 , 775 A 12 / 1987 Scott et al . 72 Claims, 19 Drawing Sheets 4 ,718 ,094 A 1 / 1988 Bahl et al. 4 ,724 , 542 A 2 / 1988 Williford 4 ,726 , 065 A 2 / 1988 Froessl 4 ,727 ,354 A 2 / 1988 Lindsay RE32 ,632 E 3 / 1988 Atkinson (51 ) Int. Ci. 4 ,736 , 296 A 4 / 1988 Katayama et al . GIOL 15 / 26 ( 2006 .01 ) 4 ,750 , 122 A 6 / 1988 Kaji et al . GIOL 15 / 28 ( 2013 . 01 ) 4 ,754 ,489 A 6 / 1988 Bokser 4 , 755 ,811 A 7 / 1988 Slavin et al. GIOL 15 / 08 (2006 .01 ) 4 ,776 ,016 A 10 / 1988 Hansen GIOL 15 / 22 ( 2006 .01 ) 4 , 783 , 804 A 11/ 1988 Juang et al . (52 ) U . S . CI. 4 , 783 , 807 A 11/ 1988 Marley CPC .. .. GIOL 15 / 26 (2013 .01 ) ; G10L 15/ 28 4 ,785 ,413 A 11/ 1988 Atsumi 4 , 790 ,028 A 12 / 1988 Ramage (2013 .01 ) ; GIOL 2015 /088 (2013 .01 ) ; GIOL 4 ,797 , 930 A 1 / 1989 Goudie 2015 / 221 ( 2013 .01 ) ; GIOL 2015 / 223 4 , 802 , 223 A 1 / 1989 Lin et al. ( 2013 .01 ) ; GIOL 2015 /225 ( 2013 .01 ) ; GIOL 4 , 803 ,729 A 2 / 1989 Baker 2015 / 228 (2013 .01 ) 4 ,807 , 752 A 2 / 1989 Chodorow 4 , 811 , 243 A 3 / 1989 Racine (58 ) Field of Classification Search 4 ,813 ,074 A 3 / 1989 Marcus CPC . GO6F 17 / 2785 ; G06F 17 / 2247; G06F 17/ 24 ; 4 , 819 , 271 A 4 / 1989 Bahl et al. G06F 17 / 241; G06F 3 / 16 ; G06F 3 /0481 ; 4 , 827 ,518 A 5 / 1989 Feustel et al . G06F 3 /0482 ; G06F 3 / 1423 ; G06F 3 /016 4 , 827 , 520 A 5 / 1989 Zeinstra See application file for complete search history . 4 ,829 ,576 A 5 / 1989 Porter 4 ,829 , 583 A 5 / 1989 Monroe et al. 4 , 831 ,551 A 5 / 1989 Schalk et al . (56 ) References Cited 4 ,833 , 712 A 5 / 1989 Bahl et al. 4 , 833 , 718 A 5 / 1989 Sprague U . S . PATENT DOCUMENTS 4 , 837 ,798 A 6 / 1989 Cohen et al . 4 , 837 , 831 A 6 / 1989 Gillick et al. 3 , 704 , 345 A 11/ 1972 Coker et al 4 , 839 , 853 A 6 / 1989 Deerwester et al . 3 ,710 , 321 A 1 / 1973 Rubenstein 4 ,852 , 168 A 7 / 1989 Sprague 3 , 828 , 132 A 8 / 1974 Flanagan et al. 4 , 862 ,504 A 8 / 1989 Nomura 3 , 979 ,557 A 9 / 1976 Schulman et al . 4 , 875 , 187 A 10 / 1989 Smith 4 ,013 , 085 A 3 / 1977 Wright 4 , 878 , 230 A 10 / 1989 Murakami et al . 4 ,081 ,631 A 3 / 1978 Feder 4 , 887 , 212 A 12 / 1989 Zamora et al . 4 , 090 ,216 A 5 / 1978 Constable 4 , 896 , 359 A 1 / 1990 Yamamoto et al . 4 , 107 , 784 A 8 / 1978 Van Bemmelen 4 , 903 , 305 A 2 / 1990 Gillick et al. 4 , 108 ,211 A 8 / 1978 Tanaka 4 , 905 , 163 A 2 / 1990 Garber et al. 4 , 159 , 536 A 6 / 1979 Kehoe et al . 4 , 908 , 867 A 3 / 1990 Silverman 4 , 181 , 821 A 1 / 1980 Pirz et al. 4 , 914 , 586 A 4 / 1990 Swinehart et al . 4 , 204 ,089 A 5 / 1980 Key et al . 4 ,914 , 590 A 4 / 1990 Loatman et al . 4 ,241 , 286 A 12 / 1980 Gordon 4 ,918 , 723 A 4 / 1990 Iggulden et al. 4 , 253 , 477 A 3 / 1981 Eichman 4 ,926 ,491 A 5 / 1990 Maeda et al. 4 ,278 , 838 A 7 / 1981 Antonov 4 ,928 ,307 A 5 / 1990 Lynn 4 , 282 ,405 A 8 / 1981 Taguchi 4 , 931 , 783 A 6 / 1990 Atkinson 4 , 310 , 721 A 1 / 1982 Manley et al . 4 ,935 ,954 A 6 / 1990 Thompson et al. 4 , 332 , 464 A 6 / 1982 Bartulis et al. 4 ,939 ,639 A 7 / 1990 Lee et al. 4 , 348 , 553 A 9 / 1982 Baker et al. 4 , 941 ,488 A 7 / 1990 Marxer et al . 4 , 384 , 169 A 5 / 1983 Mozer et al. 4 ,944 ,013 A 7 / 1990 Gouvianakis et al . 4 , 386 , 345 A 5 / 1983 Narveson et al. 4 , 945 ,504 A 7 / 1990 Nakama et al . 4 ,433 , 377 A 2 / 1984 Eustis et al . 4 ,953 , 106 A 8 / 1990 Gansner et al. 4 ,451 , 849 A 5 / 1984 Fuhrer 4 ,955 ,047 A 9 / 1990 Morganstein et al. 4 , 485 , 439 A 11/ 1984 Rothstein 4 , 965, 763 A 10 / 1990 Zamora 4 ,495 ,644 A 1 / 1985 Parks et al . 4 , 972 , 462 A 11/ 1990 Shibata 4 ,513 , 379 A 4 / 1985 Wilson et al . 4 ,974 , 191 A 11/ 1990 Amirghodsi et al . 4 ,513 , 435 A 4 / 1985 Sakoe et al. 4 ,975 , 975 A 12 / 1990 Filipski 4 ,555 , 775 A 11/ 1985 Pike 4 , 977 ,598 A 12 / 1990 Doddington et al. 4 , 577 , 343 A 3 / 1986 Oura 4 ,980 , 916 A 12 / 1990 Zinser 4 ,586 , 158 A 4 / 1986 Brandle 4 ,985 ,924 A 1 / 1991 Matsuura 4 , 587 ,670 A 5 / 1986 Levinson et al. 4 , 992 , 972 A 2 / 1991 Brooks et al. 4 , 589 ,022 A 5 / 1986 Prince et al . 4 , 994 , 966 A 2 / 1991 Hutchins 4 ,611 , 346 A 9 / 1986 Bednar et al. 4 , 994 , 983 A 4 ,615 ,081 A 10 / 1986 Lindahl 2 / 1991 Landell et al. 4 ,618 , 984 A 10 / 1986 Das et al . 5 , 001, 774 A 3 / 1991 Lee 4 ,642 , 790 A 2 / 1987 Minshull et al . 5 , 003 , 577 A 3 / 1991 Ertz et al . 4 ,653 , 021 A 3 / 1987 Takagi 5 , 007 , 095 A 4 / 1991 Nara et al. 4 ,654 , 875 A 3 / 1987 Srihari et al . 5 , 007 ,098 A 4 / 1991 Kumagai 4 ,655 , 233 A 4 / 1987 Laughlin 5 ,010 , 574 A 4 / 1991 Wang 4 ,658 ,425 A 4 / 1987 Julstrom 5 , 016 ,002 A 5 / 1991 Levanto 4 ,670 , 848 A 6 / 1987 Schramm 5 ,020 , 112 A 5 / 1991 Chou 4 ,677 , 570 A 6 / 1987 Taki 5 , 021 , 971 A 6 / 1991 Lindsay 4 ,680 ,429 A 7 / 1987 Murdock et al .