(12) United States Patent (10) Patent No.: US 9,668,024 B2 Os Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USOO9668024B2 (12) United States Patent (10) Patent No.: US 9,668,024 B2 Os et al. (45) Date of Patent: *May 30, 2017 (54) INTELLIGENT AUTOMATED ASSISTANT (58) Field of Classification Search FOR TV USER INTERACTIONS CPC combination set(s) only. See application file for complete search history. (71) Applicant: Apple Inc., Cupertino, CA (US) (56) References Cited (72) Inventors: Marcel Van Os, Cupertino, CA (US); Harry J. Saddler, Berkeley, CA (US); U.S. PATENT DOCUMENTS Lia T. Napolitano, San Francisco, CA 1,559,320 A 10, 1925 Hirsh (US); Jonathan H. Russell, Cupertino, 2,180,522 A 11/1939 Henne CA (US); Patrick M. Lister, Cupertino, (Continued) CA (US); Rohit Dasari, Cupertino, CA (US) FOREIGN PATENT DOCUMENTS CA 2666438 C 6, 2013 (73) Assignee: Apple Inc., Cupertino, CA (US) CH 681573 A5 4f1993 (*) Notice: Subject to any disclaimer, the term of this (Continued) patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days. OTHER PUBLICATIONS This patent is Subject to a terminal dis API.AI, “Android App Review—Speaktoit Assistant”. Available at claimer. <https://www.youtube.com/watch?v=myE498nyfCw>, Mar. 30. 2011, 3 pages. (21) Appl. No.: 15/085.465 (Continued) (22) Filed: Mar. 30, 2016 Primary Examiner — Kieu Oanh T Bui (74) Attorney, Agent, or Firm — Morrison & Foerster (65) Prior Publication Data LLP US 2016/0212488 A1 Jul. 21, 2016 (57) ABSTRACT Systems and processes are disclosed for controlling televi Related U.S. Application Data sion user interactions using a virtual assistant. A virtual assistant can interact with a television set-top box to control (63) Continuation of application No. 14/498.503, filed on content shown on a television. Speech input for the virtual Sep. 26, 2014, now Pat. No. 9,338.493. assistant can be received from a device with a microphone. (Continued) User intent can be determined from the speech input, and the virtual assistant can execute tasks according to the user's (51) Int. Cl. intent, including causing playback of media on the televi H04N 2L/482 (2011.01) Sion. Virtual assistant interactions can be shown on the H04N 2L/422 (2011.01) television in interfaces that expand or contract to occupy a (Continued) minimal amount of space while conveying desired informa (52) U.S. Cl. tion. Multiple devices associated with multiple displays can CPC. H04N 21/.482 (2013.01); G06F 17/30823 be used to determine user intent from speech input as well (2013.01); G09C I/00 (2013.01); as to convey information to users. In some examples, virtual (Continued) (Continued) Process 1800 Receive speech input from a user at a first - 1802 device with a first display Determine user intent based or content - 1804 displayed on the first display Determine media content based on the user - 806 intent Play the media content on a second device - 1808 associated with a second display US 9,668,024 B2 Page 2 assistant query suggestions can be provided to the user based 4,384,169 5, 1983 Mozer et al. on media content shown on a display. 4,386,345 5, 1983 Narveson et al. 4.433,377 2, 1984 Eustis et al. 4.451,849 5, 1984 Fuhrer 26 Claims, 42 Drawing Sheets 4.485439 11, 1984 Rothstein 4,495,644 1, 1985 Parks et al. 4.513,379 4, 1985 Wilson et al. 4,513.435 4, 1985 Sakoe et al. 4,555,775 11, 1985 Pike Related U.S. Application Data 4,577,343 3, 1986 Oura 4,586,158 4, 1986 Brandle (60) Provisional application No. 62/019.312, filed on Jun. 4,587,670 5, 1986 Levinson et al. 30, 2014. 4,589,022 5, 1986 Prince et al. 4,611,346 9, 1986 Bednar et al. Int. C. 4,615,081 10, 1986 Lindahl (51) 4,618,984 10, 1986 Das et al. H04N 2L/4I (2011.01) 4,642,790 2, 1987 Minshull et al. H04N 2L/443 (2011.01) 4,653,021 3, 1987 Takagi H04N 2L/858 (2011.01) 4,654,875 3, 1987 Srihari et al. H04N 2L/488 (2011.01) 4,655,233 4, 1987 Laughlin H04N 2L/84 4,658.425 4, 1987 Julstrom (2011.01) 4,670,848 6, 1987 Schramm G06F 7/30 (2006.01) 4,677,570 6, 1987 Taki GIOL I5/26 (2006.01) 4,680,429 7, 1987 Murdock et al. H04N 2L/43 (2011.01) 4,680,805 7, 1987 Scott G09C I/00 4,686,522 8, 1987 Hernandez et al. (2006.01) 4,688,195 8, 1987 Thompson et al. GIOL I5/22 (2006.01) 4,692,941 9, 1987 Jacks et al. GOL 7/00 (2013.01) 4,698.625 10, 1987 McCaskill et al. GOL 7/22 (2013.01) 4,709,390 11, 1987 Atal et al. H04N 2L/254 (2011.01) 4,713,775 12, 1987 Scott et al. H04N 2L/258 4,718,094 1, 1988 Bahl et al. (2011.01) 4,724,542 2, 1988 Williford H04N 2L/45 (2011.01) 4,726,065 2, 1988 Froessl H04N 2L/4627 (2011.01) 4,727,354 2, 1988 Lindsay HO4N 2 1/472 (2011.01) RE32,632 3, 1988 Atkinson U.S. C. 4,736,296 4, 1988 Katayama et al. (52) 4,750,122 6, 1988 Kaji et al. CPC .............. G10L 15/22 (2013.01); G10L 15/26 4,754,489 6, 1988 Bokser (2013.01); G 10L 17/005 (2013.01); G10L 4,755,811 T. 1988 Slavin et al. 17/22 (2013.01); H04N 2 1/2541 (2013.01); 4,759,070 T. 1988 Voroba et al. H04N 2 1/25891 (2013.01); H04N 21/4122 4,776,016 10, 1988 Hansen 4,783,804 11, 1988 Juang et al. (2013.01); H04N 2 1/4 126 (2013.01); H04N 4,783,807 11, 1988 Marley 21/42203 (2013.01); H04N 2 1/4312 (2013.01); 4,785,413 11, 1988 Atsumi H04N 2 1/4316 (2013.01); H04N 2 1/4438 4,790,028 12, 1988 Ramage (2013.01); H04N 2 1/4532 (2013.01); H04N 4,797,930 1, 1989 Goudie 4,802,223 1, 1989 Lin et al. 2I/4627 (2013.01); H04N 2 1/4828 (2013.01); 4,803,729 2, 1989 Baker H04N 2 1/4882 (2013.01); H04N 2 1/84 4,807,752 2, 1989 Chodorow (2013.01); H04N 2 1/858 (2013.01); GIOL 4,811,243 3, 1989 Racine 2015/223 (2013.01); H04N 2 1/47214 4,813,074 3, 1989 Marcus (2013.01) 4,819,271 4, 1989 Bahl et al. 4,827,518 5, 1989 Feustel et al. References Cited 4,827,520 5, 1989 Zeinstra (56) 4,829,576 5, 1989 Porter 4,829,583 5, 1989 Monroe et al. U.S. PATENT DOCUMENTS 4,831,551 5, 1989 Schalk et al. 4,833,712 5, 1989 Bahl et al. 2,495.222 1, 1950 Bierig 4,833,718 5, 1989 Sprague 3,704,345 1 1, 1972 Coker et al. 4,837,798 6, 1989 Cohen et al. 3,710,321 1, 1973 Rubenstein 4,837,831 6, 1989 Gillick et al. 3,787,542 1, 1974 Gallagher et al. 4,839,853 6, 1989 Deerwester et al. 3,828,132 8, 1974 Flanagan et al. 4,852,168 7, 1989 Sprague 3,979,557 9, 1976 Schulman et al. 4,862.504 8, 1989 Nomura 4,013,085 3, 1977 Wright 4,875,187 10, 1989 Smith 4,081,631 3, 1978 Feder 4,878,230 10, 1989 Murakami et al. 4,090,216 5, 1978 Constable 4,887.212 12, 1989 Zamora et al. 4,107,784 8, 1978 Van Bemmelen 4,896.359 1, 1990 Yamamoto et al. 4,108,211 8, 1978 Tanaka 2, 1990 Gillick et al. 4,159,536 6, 1979 Kehoe et al. 4,903,305 4,181,821 1, 1980 Pirz et al. 4.905,163 2, 1990 Garber et al. 4,204,089 5, 1980 Key et al. 4,908,867 3, 1990 Silverman 4,241,286 1 2, 1980 Gordon 4.914,586 4, 1990 Swinehart et al. 4,253,477 3, 1981 Eichman 4.914,590 4, 1990 Loatman et al. 4.278,838 T. 1981 Antonov 4,918,723 4, 1990 Iggulden et al. 4,282.405 8, 1981 Taguchi 4,926,491 5, 1990 Maeda et al. 4,310,721 1, 1982 Manley et al. 4,928.307 5, 1990 Lynn 4,332,464 6, 1982 Bartulis et al. 4,931,783 6, 1990 Atkinson 4,348,553 9, 1982 Baker et al. 4,935,954 6, 1990 Thompson et al. US 9,668,024 B2 Page 3 (56) References Cited 5,164,982 11, 1992 Davis 5,165,007 11, 1992 Bahl et al. U.S. PATENT DOCUMENTS 5,167,004 11, 1992 Netsch et al. 5,175,536 12, 1992 Aschliman et al. 4,939,639 7, 1990 Lee et al. 5,175,803 12, 1992 Yeh 4.941,488 7, 1990 Marxer et al. 5,175,814 12, 1992 Anick et al. 4,944,013 7, 1990 Gouvianakis et al. 5,179,627 1, 1993 Sweet et al. 4,945,504 7, 1990 Nakama et al. 5,179,652 1, 1993 Rozmanith et al. 4,953, 106 8, 1990 Gansner et al. 5,194,950 3, 1993 Murakami et al. 4,955,047 9, 1990 Morganstein et al. 5, 195,034 3, 1993 Garneau et al. 4,965,763 10, 1990 Zamora 5, 195,167 3, 1993 Bahl et al. 4,972.462 11, 1990 Shibata 5,197,005 3, 1993 Shwartz et al.