|HAI LAMA TARTANIANUS010089072B2 UN ALT MAI MULTET (12 ) United States Patent ( 10 ) Patent No. : US 10 , 089 ,072 B2 Piersol et al. (45 ) Date of Patent : Oct . 2 , 2018 ( 54 ) INTELLIGENT DEVICE ARBITRATION AND ( 56 ) References Cited CONTROL U . S . PATENT DOCUMENTS (71 ) Applicant: Apple Inc. , Cupertino , CA (US ) 1 , 559 , 320 A 10 / 1925 Hirsh 2 , 180 ,522 A 11/ 1939 Henne ( 72 ) Inventors : Kurt W . Piersol, Cupertino , CA (US ) ; 2 ,495 ,222 A 1 / 1950 Bierig Ryan M . Orr , Cupertino , CA (US ) ; 3 ,704 ,345 A 11 / 1972 Coker et al. 3 , 710 ,321 A 1 / 1973 Rubenstein Daniel J . Mandel, Tucson , AZ (US ) (Continued ) ( 73 ) Assignee : Apple Inc ., Cupertino , CA (US ) FOREIGN PATENT DOCUMENTS ( * ) Notice: Subject to any disclaimer, the term of this 2666438 C 6 / 2013 patent is extended or adjusted under 35 CH 681573 A5 4 / 1993 U . S . C . 154 (b ) by 0 days. (Continued ) ( 21 ) Appl. No. : 15 / 268, 338 OTHER PUBLICATIONS ( 22 ) Filed : Sep . 16 , 2016 International Search Report and Written Opinion received for PCT Patent Application No. PCT/ US2016 /051927 , dated Feb . 6 , 2017 , (65 ) Prior Publication Data 17 pages. US 2017 /0357478 A1 Dec . 14 , 2017 ( Continued ) Primary Examiner — Abul Azad Related U . S . Application Data ( 74 ) Attorney , Agent, or Firm — Dentons US LLP (60 ) Provisional application No . 62/ 348 , 896 , filed on Jun . (57 ) ABSTRACT 11 , 2016 . This relates to systems and processes for using a virtual assistant to arbitrate among and / or control electronic (51 ) Int. CI. devices . In one example process , a first electronic device GIOL 15 /22 ( 2006 .01 ) samples an audio input using a microphone . The first elec G06F 3 / 16 ( 2006 .01 ) tronic device broadcasts a first set of one or more values GIOL 15 / 30 ( 2013 .01 ) based on the sampled audio input . Furthermore , the first GIOL 25 /84 ( 2013. 01 ) electronic device receives a second set of one or more (52 ) U . S . CI. values , which are based on the audio input, from a second CPC . .. .. G06F 3 / 167 (2013 .01 ) ; G10L 15 / 22 electronic device . Based on the first set of one or more ( 2013. 01 ) ; GIOL 15 /30 (2013 .01 ) ; GIOL 25/ 84 values and the second set of one or more values, the first ( 2013 .01 ) ; GIOL 2015 /228 (2013 . 01 ) electronic device determines whether to respond to the audio ( 58 ) Field of Classification Search input or forego responding to the audio input. USPC . .. 704 /257 , 270 , 275 See application file for complete search history . 39 Claims, 28 Drawing Sheets USER 800 wwwwwwwwwwwwwwwwwww"Hey Siri find me a TV episode " 834 DECE wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwy DEVICE wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww DEVICE 804 ... OSNO . DEVICE 1 802 ww w wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww Awwwwwwww wwwwwwwwwwwwwwwwwwwwwwwwwwwwww . wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww vdobogor . YUNANLAR .ALAMU Wwwwwwwwwwwwww Mwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww US 10 ,089 , 072 B2 Page 2 (56 ) References Cited 4 ,831 , 551 A 5 / 1989 Schalk et al. 4 ,833 ,712 A 5 / 1989 Bahl et al. U . S . PATENT DOCUMENTS 4 , 833 , 718 A 5 / 1989 Sprague 4 , 837 ,798 A 6 / 1989 Cohen et al. 3 ,787 , 542 A 1 / 1974 Gallagher et al . 4 , 837 , 831 A 6 / 1989 Gillick et al . 3 , 828 , 132 A 8 / 1974 Flanagan et al . 4 , 839 , 853 A 6 / 1989 Deerwester et al. 3 , 979 ,557 A 9 / 1976 Schulman et al . 4 , 852 , 168 A 7 / 1989 Sprague 4 ,013 ,085 A 3 / 1977 Wright 4 , 862 , 504 A 8 / 1989 Nomura 4 ,081 ,631 A 3 / 1978 Feder 4 ,875 , 187 A 10 / 1989 Smith 4 ,090 ,216 A 5 / 1978 Constable 4 ,878 , 230 A 10 / 1989 Murakami et al. 4 , 107 , 784 A 8 / 1978 Van Bemmelen 4 , 887 , 212 A 12 / 1989 Zamora et al . 4 , 108 ,211 A 8 / 1978 Tanaka 4 , 896 , 359 A 1 / 1990 Yamamoto et al . 4 , 159 , 536 A 6 / 1979 Kehoe et al . 4 , 903 , 305 A 2 / 1990 Gillick et al. 4 , 181 , 821 A 1 / 1980 Pirz et al. 4 ,905 , 163 A 2 / 1990 Garber et al. 4 , 204 ,089 A 5 / 1980 Key et al. 4 , 908 , 867 A 3 / 1990 Silverman 4 , 241 , 286 A 12 / 1980 Gordon 4 , 914 , 586 A 4 / 1990 Swinehart et al . 4 ,253 , 477 A 3 / 1981 Eichman 4 ,914 ,590 A 4 / 1990 Loatman et al . 4 ,278 , 838 A 7 / 1981 Antonov 4 ,918 , 723 A 4 / 1990 Iggulden et al . 4 ,282 , 405 A 8 / 1981 Taguchi 4 , 926 , 491 A 5 / 1990 Maeda et al . 4 , 310 ,721 A 1 / 1982 Manley et al . 4 , 928 , 307 A 5 / 1990 Lynn 4 , 332 ,464 A 6 / 1982 Bartulis et al. 4 , 931, 783 A 6 / 1990 Atkinson 4 , 348 , 553 A 9 / 1982 Baker et al . 4 ,935 , 954 A 6 / 1990 Thompson et al. 4 , 384 , 169 A 5 / 1983 Mozer et al. 4 ,939 ,639 A 7 / 1990 Lee et al . 4 , 386 , 345 A 5 / 1983 Narveson et al . 4 ,941 ,488 A 7 / 1990 Marxer et al. 4 ,433 , 377 A 2 / 1984 Eustis et al . 4 ,944 ,013 A 7 / 1990 Gouvianakis et al. 4 ,451 , 849 A 5 / 1984 Fuhrer 4 ,945 , 504 A 7 / 1990 Nakama et al. 4 ,485 ,439 A 11/ 1984 Rothstein 4 , 953 , 106 A 8 / 1990 Gansner et al. 4 ,495 ,644 A 1 / 1985 Parks et al. 4 ,955 ,047 A 9 / 1990 Morganstein et al . 4 ,513 , 379 A 4 / 1985 Wilson et al . 4 , 965 , 763 A 10 / 1990 Zamora 4 ,513 ,435 A 4 / 1985 Sakoe et al . 4 , 972 , 462 A 11/ 1990 Shibata 4 , 555 ,775 A 11/ 1985 Pike 4 , 974 , 191 A 11/ 1990 Amirghodsi et al. 4 , 577 , 343 A 3 / 1986 Oura 4 , 975 , 975 A 12 / 1990 Filipski 4 ,586 , 158 A 4 / 1986 Brandle 4 , 977 , 598 A 12 / 1990 Doddington et al . 4 ,587 , 670 A 5 / 1986 Levinson et al. 4 , 980 , 916 A 12 / 1990 Zinser 4 ,589 ,022 A 5 / 1986 Prince et al . 4 ,985 , 924 A 1 / 1991 Matsuura 4 ,611 , 346 A 9 / 1986 Bednar et al. 4 , 992 , 972 A 2 / 1991 Brooks et al. 4 ,615 ,081 A 10 / 1986 Lindahl 4 , 994 , 966 A 2 / 1991 Hutchins 4 ,618 , 984 A 10 / 1986 Das et al . 4 , 994 , 983 A 2 / 1991 Landell et al . 4 ,642 ,790 A 2 / 1987 Minshull et al . 5 ,001 , 774 A 3 / 1991 Lee 4 ,653 ,021 A 3 / 1987 Takagi 5 ,003 , 577 A 3 / 1991 Ertz et al . 4 ,654 , 875 A 3 / 1987 Srihari et al. 5 , 007 ,095 A 4 / 1991 Nara et al . 4 ,655 , 233 A 4 / 1987 Laughlin 5 , 007 ,098 A 4 / 1991 Kumagai 4 ,658 ,425 A 4 / 1987 Julstrom 5 ,010 , 574 A 4 / 1991 Wang 4 ,670 , 848 A 6 / 1987 Schramm 5 ,016 , 002 A 5 / 1991 Levanto 4 ,677 , 570 A 6 / 1987 Taki 5 , 020 , 112 A 5 / 1991 Chou 4 ,680 ,429 A 7 / 1987 Murdock et al . 5 , 021 , 971 A 6 / 1991 Lindsay 4 ,680 , 805 A 7 / 1987 Scott 5 , 022 ,081 A 6 / 1991 Hirose et al. 4 ,686 , 522 A 8 / 1987 Hernandez et al . 5 ,027 , 110 A 6 / 1991 Chang et al. 4 ,688 , 195 A 8 / 1987 Thompson et al. 5 ,027 ,406 A 6 / 1991 Roberts et al . 4 ,692 , 941 A 9 / 1987 Jacks et al . 5 ,027 ,408 A 6 / 1991 Kroeker et al. 4 ,698 ,625 A 10 / 1987 McCaskill et al. 5 ,029 ,211 A 7 / 1991 Ozawa 4 ,709 ,390 A 11/ 1987 Atal et al. 5 , 031, 217 A 7 / 1991 Nishimura 4 ,713 , 775 A 12 / 1987 Scott et al . 5 ,032 , 989 A 7 / 1991 Tornetta 4 , 718 ,094 A 1 / 1988 Bahl et al. 5 ,033 ,087 A 7 / 1991 Bahl et al. 4 ,724 , 542 A 2 / 1988 Williford 5 ,040 , 218 A 8 / 1991 Vitale et al. 4 , 726 ,065 A 2 / 1988 Froessl 5 , 046 , 099 A 9 / 1991 Nishimura 4 , 727 , 354 A 2 / 1988 Lindsay 5 , 047 ,614 A 9 / 1991 Bianco RE32 ,632 E 3 / 1988 Atkinson 5 ,047 ,617 A 9 / 1991 Shepard et al. 4 ,736 ,296 A 4 / 1988 Katayama et al. 5 ,050 , 215 A 9 / 1991 Nishimura 4 , 750 , 122 A 6 / 1988 Kaji et al. 5 ,053 , 758 A 10 / 1991 Cornett et al . 4 , 754 , 489 A 6 / 1988 Bokser 5 , 054 ,084 A 10 / 1991 Tanaka et al . 4 , 755 , 811 A 7 / 1988 Slavin et al. 5 ,057 ,915 A 10 / 1991 Von Kohorn 4 ,759 , 070 A 7 / 1988 Voroba et al . 5 ,062 , 143 A 10 / 1991 Schmitt 4 ,776 , 016 A 10 / 1988 Hansen 5 , 067 , 158 A 11/ 1991 Arjmand 4 , 783 , 804 A 11/ 1988 Juang et al . 5 , 067 ,503 A 11/ 1991 Stile 4 ,783 , 807 A 11 / 1988 Marley 5 , 072 ,452 A 12 / 1991 Brown et al . 4 ,785 ,413 A 11/ 1988 Atsumi 5 ,075 , 896 A 12 / 1991 Wilcox et al . 4 ,790 ,028 A 12 / 1988 Ramage 5 ,079 ,723 A 1 / 1992 Herceg et al . 4 , 797 , 930 A 1 / 1989 Goudie 5 ,083 , 119 A 1 / 1992 Trevett et al . 4 , 802 , 223 A 1 / 1989 Lin et al . 5 , 083 , 268 A 1 / 1992 Hemphill et al . 4 , 803 , 729 A 2 / 1989 Baker 5 , 086 ,792 A 2 / 1992 Chodorow 4 , 807 , 752 A 2 / 1989 Chodorow 5 ,090 ,012 A 2 / 1992 Kajiyama et al .