USOO931 8108B2

(12) United States Patent (10) Patent No.: US 9,318,108 B2 Gruber et al. (45) Date of Patent: Apr. 19, 2016

(54) INTELLIGENT AUTOMATED ASSISTANT G06F 17/30867; G06F 17/3087; G06F 17/30879; G06F 3/038; G10L 15/181; G 10L (75) Inventors: Thomas Robert Gruber, Emerald Hills, 15/19, G10L 15/30; G 10L 15/18: G 10L CA (US); Adam John Cheyer, Oakland, 15/1822; G 10L 15/20: G 10L 15/24: G10L CA (US); Dag Kittlaus, San Jose, CA 15/26 (US); Didier Rene Guzzoni, USPC ...... 704/270. 1, 275,235, 9, 10, 270; Mont-sur-Rolle (CH); Christopher 706/11: 707/909.01, 999.002; 705/28; Dean Brigham, San Jose, CA (US); 709/224, 217: 715/234, 727, 764,728; Richard Donald Giuli, Arroyo Grande, 379/201.01, 230; 455/556.1 CA (US); Marcello Bastea-Forte, New See application file for complete search history. York, NY (US); Harry Joseph Saddler, Berkeley, CA (US) (56) References Cited U.S. PATENT DOCUMENTS (73) Assignee: Apple Inc., Cupertino, CA (US) 1,559,320 A 10, 1925 Hirsh (*) Notice: Subject to any disclaimer, the term of this 2,180,522 A 11/1939 Henne patent is extended or adjusted under 35 U.S.C. 154(b) by 441 days. (Continued) FOREIGN PATENT DOCUMENTS (21) Appl. No.: 12/987,982 CA 2666438 C 6, 2013 (22) Filed: Jan. 10, 2011 CH 681573 A5 4f1993 Prior Publication Data (Continued) (65) OTHER PUBLICATIONS US 2012/OO16678A1 Jan. 19, 2012 Australian Office Action dated Nov. 13, 2012 for Application No. Related U.S. Application Data 2011205426, 7 pages. (Continued) (60) Provisional application No. 61/295,774, filed on Jan. 18, 2010. Primary Examiner — Vijay B Chawan (74) Attorney, Agent, or Firm — Morrison & Foerster LLP (51) Int. C. GIOL 2L/00 (2013.01) (57) ABSTRACT GIOL 5/8 (2013.01) An intelligent automated assistant system engages with the (Continued) user in an integrated, conversational manner using natural language dialog, and invokes external services when appro (52) U.S. C. priate to obtain information or perform various actions. The CPC ...... G10L 15/1815 (2013.01); G06F 3/167 system can be implemented using any of a number of differ (2013.01); G06F 9/54 (2013.01); ent platforms, such as the web, email. Smartphone, and the (Continued) like, or any combination thereof. In one embodiment, the (58) Field of Classification Search system is based on sets of interrelated domains and tasks, and CPC ...... H04M 1/72561; H04M 2250/74; employs additional functionally powered by external services H04M 3/4938; G06F 17/30731: G06F with which the system can interact. 17/30241; G06F 17/30256; G06F 17/30265; 48 Claims, 47 Drawing Sheets

102 intelligent Automated Assistant

Active Input Short Term Long Tem Elicitation Personal Personal i094 emory 052 Memory iO54

Domain Modeis 1086 Qutput to User input User 004 Wocatulary 008 158 Ontology Dialog Flow 1050 Models Language 1087 Pattern Recognizers 1060 Other / Other Events'Facts Dialog Flow Actions 1008 010 Domain Processor Entity 1080 Databases Output 1072 Processor

Services 1090 Orchestration Language 1082 Interpreter 1073 Services 1084 US 9,318.108 B2 Page 2

(51) Int. Cl. 4,776,016 A 10/1988 Hansen G06F 7/30 2006.O1 4,783,804 A 11/1988 Juang et al. GIOL I5/22 ( ) 4,783,807 A 1 1/1988 Marley (2006.01) 4,785,413. A 1 1/1988 Atsumi G06F 7/28 (2006.01) 4,790,028 A 12/1988 Ramage

G06F 3/16 (2006.01) 3.3w W4- A 3. SudiIn et al. G06F 9/54 (2006.01) 4,803,729 A 2/1989 Baker GIOL I5/26 (2006.01) 4,807,752 A 2/1989 Chodorow GIOL 2/06 (2013.01) 4,811,243 A 3, 1989 Racine 4,813,074. A 3/1989 Marcus (52) U.S. Cl. 4,819,271. A 4, 1989 Bahl et al. CPC ...... G06F 17/28 (2013.01); G06F 17/3087 4,827,518 A 5/1989 Feustel et al. (2013.01); G 10L 15/22 (2013.01); G 10L 15/26 4,827,520 A 5/1989 Zeinstra (2013.01); G 10L 2 1/06 (2013.01) 4,829,576 A 5/1989 Porter 4,829,583 A 5/1989 Monroe et al. 4,831,551 A 5/1989 Schalk et al. (56) References Cited 4,833,712 A 5/1989 Bahl et al. 4,833,718 A 5/1989 Sprague U.S. PATENT DOCUMENTS 4,837.798. A 6/1989 Cohen et al. 4,837,831. A 6/1989 Gillicket al. 3,704,345 A 11/1972 Coker et al. 4,839,853. A 6/1989 Deerwester et al. 3,710,321 A 1/1973 Rubenstein 4,852,168 A 7, 1989 Sprague 3,979,557 A 9, 1976 Schulman et al. 4875.87 A 10, 1989 Smith 4,013,085 A 3/1977 Wright 4,878,230 A 10/1989 Murakami et al. 4,081,631 A 3/1978 Feder 4,887.212 A 12/1989 Zamora et al. 4,107,784.2.87 A 8/1978ERA, Van Bemmelen 4,903,3054,896,359 A 2/19901/1990 YamamotoGillicket al. et al. 4,108,211 A 8/1978 Tanaka 4.905,163 A 2/1990 Garber et al. 4,181,821.;: A $21/1980 Pirzbit et al. al. 4.914,5864,908,867 A 4/19903/1990 SilvermanSwinehart et al. 4.204,089 A 5/1980 Key et al. 4.914,590 A 4/1990 Loatman et al. 4,241.286 A 12, 1980 Gordon 4,918,723. A 4/1990 Iggulden et al. 4,253,477 A 3/1981 Eichman 4,926,491. A 5/1990 Maeda et al. 4.278,838 A 7/1981 Antonov 4,928.307 A 5/1990 Lynn 4,282.405 A 8/1981 Taguchi 4,931,783. A 6/1990 Atkinson 4,310,721. A 1/1982 Manley et al. 4,935,954. A 6/1990 Thompson et al. 4,332,464 A 6/1982 Bartulis et al. 4,939,639 A 7/1990 Lee et al. 4,348,553 A 9, 1982 Baker et al. 4.941,488 A 7/1990 Marxer et all 4,384,169 A 5/1983 Mozer et al. 4,944,013- I k A 7, 1990 Gouvianakis. . . et al. 4,386,345 A 5, 1983 Narveson et al. 4,945,504. A 7/1990 Nakama et al. 4.433,377 A 2f1984 Eustis et al. 4,953,106 A 8, 1990 Gansner et al. 4,451,849. A 5/1984 Fuhrer 4.955,047 A 9/1990 Morganstein et al. 4,485.439 A 1 1/1984 Rothstein 4,965,763 A 10/1990 Zamora 4,495,644. A 1/1985 Parks et al. 4.972.462. A 11, 1990 Shibata 4.513,379 A 4, 1985 Wilson et al. 4,974,191- 4 A 11/1990 Amirghodsi et al. 4,555,775343. A 1 1/1985 PikeSeet al. 4,977.598.4,975,975 A 12/1990 FilipskiDoddington et al. 4,577.343 A 3/1986 Oura 4,980,916 A 12/1990 Zinser 4,586, 158. A 4, 1986 Brandle 4,985,924. A 1/1991 Matsuura 4,587,670 A 5, 1986 Levinson et al. 4992.972 A 2f1991 Brooks etal 4,589,022 A 5, 1986 Prince et al. 4,994,966 A 2f1991 Hutchins 4,611,3464,615,081. A 10,9/1986 1986 BednarLindahl et al. 4,994,983- - I A 2/1991 Landell et al. 4,618,984 A 10, 1986 Das et al. 387, A. E. E. ea 4,653,021 A 3/1987 Takagi WW t 4,654,875. A 3/1987 Srihari et al. 3.999; A 23: sial 4,658.425. A 4, 1987 Julstrom 5.020.112 A 5/199 Chou 4,670,848. A 6/1987 Schramm 5.021971. A 6/1991 Lindsay 4,677,570 A 6/1987 Taki 5,022,081. A 6/1991 Hirose et al. 2.85 A 23, Milocket al. 5,027,1 10 A 6/1991 Chang et al. 4,686,522s sw A 8/1987 Hernandez et al. 5,027.4068578. A 6/199136 E.Roberts et al. 4,688,195 A 8, 1987 Thompson et al. 5029.3 A 7, 1991 Ozawa 4,692,941. A 9, 1987 Jacks et al. 5,031.217. A 7/1991 Nishimura 4,698.625. A 10/1987 McCaskill et al. 5,032.989 A 7/1991 Tornetta 4,709,390 A 1 1/1987 Atal et al. 5,033,087. A 7/1991 Bahletal 4,713,775 A 12/1987 Scott et al. 5,040.2is A 81991 Vitale etal 4,718,094. A 1/1988 Bahl et al. w w itale et al. 4724.543 A 2, 1988 Williford 5,046,099 A 9, 1991 Nishimura 4,726,065 A 2/1988 Froessl 5,047,614 A 9/1991 Bianco 4.727,354 A 2, 1988 Lindsay 5,047,617 A 9, 1991 Shepard et al. RE32,633 E 3, 1988 William 5,050,215. A 9/1991 Nishimura 4,736,296 A 4, 1988 Katayama et al. 5,053,758 A 10, 1991 Cornett et al. 4,750,122 A 6/1988 Kaji et al. 5,054,084 A 10/1991 Tanaka et al. 4,754,489 A 6/1988 Bokser 5,057,915 A 10/1991 Kohorn et al. 4,755,811 A 7/1988 Slavin et al. 5,067,158 A 1 1/1991 Arjmand US 9,318.108 B2 Page 3

(56) References Cited 5,282,265 1, 1994 Rohra Suda et al. 5,283,818 2, 1994 Klausner et al. U.S. PATENT DOCUMENTS 5,287,448 2, 1994 Nicol et al. 5,289,562 2, 1994 Mizuta et al. 5,067,503 11, 1991 Stile RE34,562 3, 1994 Murakami et al. 5,072,452 12, 1991 Brown et al. 5,291,286 3, 1994 Murakami et al. 5,075,896 12, 1991 Wilcox et al. 5,293,254 3, 1994 Eschbach 5,079,723 1, 1992 Herceg et al. 5,293,448 3, 1994 Honda 5,083,119 1, 1992 Trevett et al. 5,293,452 3, 1994 Picone et al. 5,083,268 1, 1992 Hemphill et al. 5,296,642 3, 1994 Konishi 5,086,792 2, 1992 Chodorow 5,297,170 3, 1994 Eyuboglu et al. 5,090,012 2, 1992 Kajiyama et al. 5,297,194 3, 1994 Hunt et al. 5,091,790 2, 1992 Silverberg 5,299,125 3, 1994 Baker et al. 5,091,945 2, 1992 Kleijn 5,299,284 3, 1994 Roy 5,103,498 4, 1992 Lanier et al. 5,301,109 4, 1994 Landauer et al. 5,109,509 4, 1992 Katayama et al. 5,303,406 4, 1994 Hansen et al. 5,111,423 5, 1992 Kopec, Jr. et al. 5,305,205 4, 1994 Weber et al. 5,119,079 6, 1992 Hube et al. 5,305,768 4, 1994 Gross et al. 5,122,951 6, 1992 Kamiya 5,309,359 5, 1994 Katz et al. 5,123,103 6, 1992 Ohtaki et al. 5,315,689 5, 1994 Kanazawa et al. 5,125,022 6, 1992 Hunt et al. 5,317,507 5, 1994 Gallant 5,125,030 6, 1992 Nomura et al. 5,317,647 5, 1994 Pagallo 5,127,043 6, 1992 Hunt et al. 5,325,297 6, 1994 Bird et al. 5,127,053 6, 1992 Koch 5,325,298 6, 1994 Gallant 5,127,055 6, 1992 Larkey 5,325,462 6, 1994 Farrett 5,128,672 7, 1992 Kaehler 5,326,270 T/1994 Ostby et al. 5,133,011 7, 1992 McKiel, Jr. 5,327,342 T/1994 Roy 5,133,023 7, 1992 Bokser 5,327.498 T/1994 Hamon 5,142.584 8, 1992 Ozawa 5,329,608 T/1994 Bocchieri et al. 5,144,875 9, 1992 Nakada 5,333,236 T/1994 Bahl et al. 5,148,541 9, 1992 Lee et al. 5,333,266 T/1994 Boaz et al. 5,153,913 10, 1992 Kandefer et al. 5,333,275 T/1994 Wheatley et al. 5,157,610 10, 1992 Asano et al. 5,335,011 8, 1994 Addeo et al. 5,157,779 10, 1992 Washburn et al. 5,335,276 8, 1994 Thompson et al. 5,161,102 11, 1992 Griffin et al. 5,341,293 8, 1994 Vertelney et al. 5,164,900 11, 1992 Bernath 5,341,466 8, 1994 Perlin et al. 5,164,982 11, 1992 Davis 5,345,536 9, 1994 Hoshimi et al. 5,165,007 11, 1992 Bahl et al. 5,349,645 9, 1994 Zhao 5,167,004 11, 1992 Netsch et al. 5,353,374 10, 1994 Wilson et al. 5,175,536 12, 1992 Aschliman et al. 5,353,376 10, 1994 Oh et al. 5,175,803 12, 1992 Yeh 5,353,377 10, 1994 Kuroda et al. 5,175,814 12, 1992 Anicket al. 5,353,408 10, 1994 Kato et al. 5,179,627 1, 1993 Sweet et al. 5,353,432 10, 1994 Richek et al. 5,179,652 1, 1993 Rozmanith et al. 5,357,431 10, 1994 Nakada et al. 5,194,950 3, 1993 Murakami et al. 5,367,640 11, 1994 Hamilton et al. 5, 195,034 3, 1993 Garneau et al. 5,369,575 11, 1994 Lamberti et al. 5, 195,167 3, 1993 Bahl et al. 5,369,577 11, 1994 Kadashevich et al. 5,197,005 3, 1993 Shwartz et al. 5,371,853 12, 1994 Kao et al. 5, 199,077 3, 1993 Wilcox et al. 5,371,901 12, 1994 Reed et al. 5,201,034 4, 1993 Matsuura et al. 5,373,566 12, 1994 Murdock 5,202,952 4, 1993 Gillick et al. 5,377,103 12, 1994 Lamberti et al. 5,208,862 5, 1993 Ozawa 5,377,301 12, 1994 Rosenberg et al. 5,210,689 5, 1993 Baker et al. 5,377,303 12, 1994 Firman 5,212,638 5, 1993 Bernath 5,384,671 1/1995 Fisher 5,212,821 5, 1993 Gorin et al. 5,384,892 1/1995 Strong 5,216,747 6, 1993 Hardwicket al. 5,384,893 1/1995 Hutchins 5,218,700 6, 1993 Beechick 5,386,494 1/1995 White 5,220,629 6, 1993 Kosaka et al. 5,386.556 1/1995 Hedin et al. 5,220,639 6, 1993 Lee 5,390,236 2, 1995 Klausner et al. 5,220,657 6, 1993 Bly et al. 5,390,279 2, 1995 Strong 5,222,146 6, 1993 Bahl et al. 5,390,281 2, 1995 Luciw et al. 5,230,036 7, 1993 Akamine et al. 5,392,419 2, 1995 Walton 5,231,670 7, 1993 Goldhor et al. 5,396,625 3, 1995 Parkes 5,235,680 8, 1993 Bijnagte 5,400,434 3, 1995 Pearson 5,237,502 8, 1993 White et al. 5,404,295 4, 1995 Katz et al. 5,241,619 8, 1993 Schwartz et al. 5,406,305 4, 1995 Shimomura et al. 5,252.951 10, 1993 Tannenbaum et al. 5,408,060 4, 1995 Muurinen 5,253,325 10, 1993 Clark 5,412,756 5, 1995 Bauman et al. 5,257,387 10, 1993 Richek et al. 5,412,804 5, 1995 Krishna 5,260,697 11, 1993 Barrett et al. 5,412,806 5, 1995 Du et al. 5,266,931 11, 1993 Tanaka 5,418,951 5, 1995 Damashek 5,266,949 11, 1993 Rossi 5,422,656 6, 1995 Allard et al. 5,267,345 11, 1993 Brown et al. 5,424,947 6, 1995 Nagao et al. 5,268,990 12, 1993 Cohen et al. 5.425, 108 6, 1995 Hwang et al. 5,274,771 12, 1993 Hamilton et al. 5,428,731 6, 1995 Powers, III 5,274,818 12, 1993 Vasilevsky et al. 5.434,777 7, 1995 Luciw 5,276,616 1, 1994 Kuga et al. 5.440,615 8, 1995 Caccuro et al. 5,276,794 1, 1994 Lamb, Jr. 5,442,598 8, 1995 Haikawa et al. 5,278.980 1, 1994 Pedersen et al. 5,442,780 8, 1995 Takanashi et al. US 9,318.108 B2 Page 4

(56) References Cited 5,596,994 A 1, 1997 Bro 5,608,624 A 3, 1997 Luciw U.S. PATENT DOCUMENTS 5,608,698 A 3, 1997 Yamanoi et al. 5,608,841 A 3, 1997 Tsuboka 5,444,823 A 8/1995 Nguyen 5,610,812 A 3, 1997 Schabes et al. 5.449,368 A 9, 1995 Kuzmak 5,613,036 A 3/1997 Strong 5.450,523 A 9, 1995 Zhao 5,613,122 A 3, 1997 Burnard et al. 5.455.8ss A 10/1995 Iyengar et al. 5,615,378 A 3, 1997 Nishino et al. 5,457,768 A 10, 1995 Tsuboi et al. 5,615,384 A 3, 1997 Allard et al. 5,459,488 A 10/1995 Geiser 5,616,876 A 4, 1997 Cluts 5,463,696 A 10, 1995 Beernink et al. 5,617,386 A 4, 1997 Choi 5,463,725 A 10, 1995 Henckel et al. 5,617,507 A 4, 1997 Lee et al. 5,465,401 A 1 1/1995 Thompson 5,617,539 A 4/1997 Ludwig et al. 5,469,529 A 1 1/1995 Bimbot et al. 5,619,583 A 4, 1997 Page et al. 5,471,611 A 1 1/1995 McGregor 5,619,694 A 4/1997 Shimazu 5,473,728 A 12/1995 Luginbuhl et al. 5,621,859 A 4, 1997 Schwartz et al. 5,475,587. A 12/1995 Anicket al. 5,621,903. A 4/1997 Luciw et al. 5,475,796 A 12/1995 Iwata 5,627,939 A 5/1997 Huang et al. 5,477.447 A 12/1995 Luciw et al. 5,634,084 A 5/1997 Malsheen et al. 5,477,448 A 12/1995 Golding et al. 5,636,325 A 6, 1997 Farrett 5,477,451 A 12/1995 Brown et al. 5,638,425 A 6/1997 Meador, III et al. 5,479,488 A 12/1995 Lennig et al. 5,638.489 A 6/1997 Tsuboka 5,481,739 A 1/1996 Staats 5,638,523 A 6/1997 Mullet et al. 5,483.261 A 1/1996 Yasutake 5,640.487 A 6/1997 Lau et al. 5,485,372 A 1/1996 Golding et al. 5,642.464 A 6/1997 Yue et al. 5,485,543 A 1/1996 ASO 5,642,466 A 6/1997 Narayan 5.488.204 A 1/1996 Mead et al. 5,642,519 A 6, 1997 Martin 5.488,727 A 1/1996 Agrawal et al. 5,644,656 A 7/1997 Akra et al. 5.490,234 A 2/1996 Narayan 5,644,727 A 7/1997 Atkins 5,491,758 A 2/1996 Bellegarda et al. 5,644,735 A 7/1997 Luciw et al. 5,491,772 A 2f1996 Hardwicket al. 5,649,060 A 7/1997 Ellozy et al. 5,493,677 A 2/1996 Balogh 5,652,828 A 7/1997 Silverman 5,495,604 A 2/1996 Harding et al. 5,652,884 A 7, 1997 Palevich 5,497.319 A 3/1996 Chong et al. 5,652,897 A 7/1997 Linebarger et al. 5,500,903. A 3, 1996 Gulli 5,661,787 A 8, 1997 Pocock 5,500,905 A 3, 1996 Martin et al. 5,664,055 A 9, 1997 Kroon 5,500,937 A 3/1996 Thompson-Rohrlich 5,670.985 A 9/1997 Cappels, Sr. et al. 5,502,774 A 3/1996 Bellegarda et al. 5,675,819 A 10, 1997 Schuetze 5,502,790 A 3, 1996 Y 5,682,475 A 10, 1997 Johnson et al. 5,502,791 A 3, 1996 Nishimura et al. 5,682,539 A 10/1997 Conrad et al. 5,515,475 A 5/1996 Gupta et al. 5,684.513 A 11/1997 Decker 5,521,816 A 5, 1996 Roche et al. 5,687,077 A 1 1/1997 Gough, Jr. 5,524,140 A 6, 1996 Klausner et al. 5,689.287 A 1 1/1997 Mackinlay et al. 5,533,182 A 7, 1996 Bates et al. 5,689,616 A 11/1997 Li 5.535,121 A 7, 1996 Roche et al. 5,689,618 A 1 1/1997 Gasper et al. 5,536,902 A 7, 1996 Serra et al. 5,692.205 A 11/1997 Berry et al. 5,537,317 A 7, 1996 Schabes et al. 5,696,962 A 12/1997 Kupiec 5,537,618 A 7, 1996 Boulton et al. 5,699,082 A 12/1997 Marks et al. 5,537,647 A 7/1996 Hermansky et al. 5,701,400 A 12/1997 Amado 5.543,588 A 8, 1996 Bisset et al. 5,706,442 A 1/1998 Anderson et al. 5.543,897 A 8/1996 Altrieth, III 5,708.659 A 1/1998 Rostoker et al. 5,544,264 A 8/1996 Bellegarda et al. 5,708,822 A 1/1998 Wical 5,548,507 A 8, 1996 Martino et al. 5,710,886 A 1/1998 Christensen et al. 5,555.343 A 9, 1996 Luther 5,710,922 A 1/1998 Alley et al. 5,555,344 A 9, 1996 Zunkler 5,712,949 A 1/1998 Kato et al. 5,559,301 A 9/1996 Bryan, Jr. et al. 5,712,957 A 1/1998 Waibel et al. 5,559,945 A 9, 1996 Beaudet et al. 5,715,468 A 2f1998 Budzinski 5,564,446 A 10, 1996 Wiltshire 5,717,877 A 2f1998 Orton et al. 5,565,888 A 10, 1996 Selker 5,721,827 A 2, 1998 Logan et al. 5,568,536 A 10, 1996 Tiller et al. 5,721,949 A 2f1998 Smith et al. 5,568,540 A 10, 1996 Greco et al. 5,724.406 A 3, 1998 Juster 5,570,324 A 10, 1996 Geil 5,724,985 A 3, 1998 Snell et al. 5,572,576 A 11/1996 Klausner et al. 5,726,672 A 3, 1998 Hernandez et al. 5,574,823. A 1 1/1996 Hassanein et al. 5,727,950 A 3/1998 Cook et al. 5,574,824 A 1 1/1996 Slyhet al. 5,729,694 A 3, 1998 Holzrichter et al. 5,577,135 A 1 1/1996 Grajski et al. 5,729,704 A * 3/1998 Stone et al...... T15,804 5,577,164 A 1 1/1996 Kaneko et al. 5,732,216 A 3/1998 Logan et al. 5,577,241 A 1 1/1996 Spencer 5,732,390 A 3/1998 Katayanagi et al. 5,578,808 A 1 1/1996 Taylor 5,732,395 A 3, 1998 Silverman 5,579,037 A 11/1996 Tahara et al. 5,734,750 A 3, 1998 Arai et al. 5,579,436 A 11/1996 Chou et al. 5,734,791 A 3, 1998 Acero et al. 5,581,484 A 12, 1996 Prince 5,736,974 A 4/1998 Selker 5,581,652 A 12/1996 Abe et al. 5,737.487 A 4/1998 Bellegarda et al. 5,581,655 A 12/1996 Cohen et al. 5,737,609 A 4/1998 Reed et al. 5,583,993 A 12/1996 Foster et al. 5,737,734. A 4/1998 Schultz 5,584,024 A 12/1996 Shwartz 5,739,451 A 4/1998 Winksy et al. 5,594,641 A 1/1997 Kaplanet al. 5,740,143 A 4/1998 Suetomi 5,596.260 A 1/1997 Moravec et al. 5,742,705 A 4/1998 Parthasarathy 5,596,676 A 1/1997 Swaminathan et al. 5,742,736 A 4/1998 Haddock US 9,318.108 B2 Page 5

(56) References Cited 5,864,806 1/1999 Mokbel et al. 5,864,815 1/1999 Rozak et al. U.S. PATENT DOCUMENTS 5,864,844 1/1999 James et al. 5,864,855 1/1999 Ruocco et al. 5,745,116 4, 1998 Pisutha-Arnond 5,864,868 1/1999 Contois 5,745,873 4, 1998 Braida et al. 5,867,799 2, 1999 Lang et al. 5,748,512 5, 1998 Vargas 5,870,710 2, 1999 Ozawa et al. 5,748,974 5, 1998 Johnson 5,873,056 2, 1999 Liddy et al. 5,749,071 5, 1998 Silverman 5,875.427 2, 1999 Yamazaki 5,749,081 5, 1998 Whiteis 5,875.429 2, 1999 Douglas 5,751,906 5, 1998 Silverman 5,875.437 2, 1999 Atkins 5,757,358 5, 1998 Osga 5,876,396 3, 1999 Lo et al. 5,757,979 5, 1998 Hongo et al. 5,877,751 3, 1999 Kanemitsu et al. 5,758,079 5, 1998 Ludwig et al. 5,877,757 3, 1999 Baldwin et al. 5,758,083 5, 1998 Singh et al. 5,878,393 3, 1999 Hata et al. 5,758,314 5, 1998 McKenna 5,878,394 3, 1999 Muhling 5,759,101 6, 1998 Von Kohorn 5,878,396 3, 1999 Henton 5,761,640 6, 1998 KalyanSwamy et al. 5,880,411 3, 1999 Gillespie et al. 5,765,131 6, 1998 Stentiford et al. 5,880,731 3, 1999 Liles et al. 5,765,168 6, 1998 Burrows 5,884,039 3, 1999 Ludwig et al. 5,771.276 6, 1998 Wolf 5,884,323 3, 1999 Hawkins et al. 5,774,834 6, 1998 Visser 5,890,117 3, 1999 Silverman 5,774,855 6, 1998 Foti et al. 5,890,122 3, 1999 Van Kleeck et al. 5,774,859 6, 1998 Houser et al. 5,891, 180 4, 1999 Greeninger et al. 5,777,614 7, 1998 Ando et al. 5,893,126 4, 1999 Drews et al. 5,778.405 7, 1998 Ogawa 5,893,132 4, 1999 Huffman et al. 5,790,978 8, 1998 Olive et al. 5,895.448 4, 1999 Vysotsky et al. 5,794,050 8, 1998 Dahlgren et al. 5,895,464 4, 1999 Bhandari et al. 5,794, 182 8, 1998 Manduchi et al. 5,895,466 4, 1999 Goldberg et al. 5,794,207 8, 1998 Walker et al. 5,896,321 4, 1999 Miller et al. 5,794,237 8, 1998 Gore, Jr. 5,896,500 4, 1999 Ludwig et al. 5,797,008 8, 1998 Burrows 5,899,972 5, 1999 Miyazawa et al. 5,799,268 8, 1998 Boguraev 5,905,498 5, 1999 Diament et al. 5,799,269 8, 1998 Schabes et al. 5,909,666 6, 1999 Gould et al. 5,799.276 8, 1998 Komissarchik et al. 5,912.951 6, 1999 Checchio et al. 5,801,692 9, 1998 Muzio et al. 5,912,952 6, 1999 Brendzel 5,802.466 9, 1998 Gallant et al. 5,913, 193 6, 1999 Huang et al. 5,802.526 9, 1998 Fawcett et al. 5,915.001 6, 1999 Uppaluru et al. 5,812,697 9, 1998 Sakai et al. 5,915,236 6, 1999 Gould et al. 5,812,698 9, 1998 Platt et al. 5,915,238 6, 1999 Tjaden 5,815,142 9, 1998 Allard et al. 5,915,249 6, 1999 Spencer 5,815,225 9, 1998 Nelson 5.917487 6, 1999 Ulrich 5,818, 142 10, 1998 Edleblute et al. 5,918,303 6, 1999 Yamaura et al. 5,818.451 10, 1998 Bertram et al. 5,920,327 7, 1999 Seidensticker, Jr. 5,818,924 10, 1998 King et al. 5,920,836 7, 1999 Gould et al. 5,822.288 10, 1998 Shinada 5,920,837 7, 1999 Gould et al. 5,822,720 10, 1998 Bookman et al. 5,923,757 7, 1999 Hocker et al. 5,822,730 10, 1998 Roth et al. 5,924,068 7, 1999 Richard et al. 5,822,743 10, 1998 Gupta et al. 5,926,769 7, 1999 Valimaa et al. 5,825,349 10, 1998 Meier et al. 5,926,789 7, 1999 Barbara et al. 5,825,352 10, 1998 Bisset et al. 5,930,408 7, 1999 Seto 5,825,881 10, 1998 Colvin, Sr. 5,930,751 7, 1999 Cohrs et al. 5,826.261 10, 1998 Spencer 5,930,754 7, 1999 Karaali et al. 5,828,768 10, 1998 Eatwell et al. 5,930,769 7, 1999 Rose 5,828,999 10, 1998 Bellegarda et al. 5,930.783 7, 1999 Li et al. 5,832,433 11, 1998 Yashchin et al. 5,933,477 8, 1999 Wu 5,832,435 11, 1998 Silverman 5,933,806 8, 1999 Beyerlein et al. 5,833,134 11, 1998 Ho et al. 5,933,822 8, 1999 Braden-Harder et al. 5,835.077 11, 1998 Dao et al. 5,936,926 8, 1999 Yokouchi et al. 5,835,079 11, 1998 Shieh 5.937,163 8, 1999 Lee et al. 5,835,721 11, 1998 Donahue et al. 5,940,811 8, 1999 Norris 5,835,732 11, 1998 Kikinis et al. 5,940,841 8, 1999 Schmucket al. 5,835,893 11, 1998 Ushioda 5,941,944 8, 1999 Messerly 5,839,106 11, 1998 Bellegarda 5,943,043 8, 1999 Furuhata et al. 5,841,902 11, 1998 Tu 5,943,049 8, 1999 Matsubara et al. 5,842,165 11, 1998 Raman et al. 5,943,052 8, 1999 Allen et al. 5,845,255 12, 1998 Mayaud 5,943,429 8, 1999 Haendel et al. 5,848,410 12, 1998 Walls et al. 5,943,443 8, 1999 Itonori et al. 5,850,480 12, 1998 Scanlon 5,943,670 8, 1999 Prager 5,850,629 12, 1998 Holm et al. 5,946,647 8, 1999 Miller et al. 5,854,893 12, 1998 Ludwig et al. 5.948,040 9, 1999 DeLorime et al. 5,855,000 12, 1998 Waibel et al. 5,949,961 9, 1999 Sharman 5,857, 184 1, 1999 Lynch 5,950,123 9, 1999 Schwelb et al. 5,859,636 1, 1999 Pandit 5,952.992 9, 1999 Helms 5,860,063 1, 1999 Gorin et al. 5,953,541 9, 1999 King et al. 5,860,064 1, 1999 Henton 5,956,021 9, 1999 Kubota et al. 5,860,075 1, 1999 Hashizume et al. 5,956,699 9, 1999 Wong et al. 5,862,223 1, 1999 Walker et al. 5,960,394 9, 1999 Gould et al. 5,862,233 1, 1999 Poletti 5,960,422 9, 1999 Prasad US 9,318.108 B2 Page 6

(56) References Cited 6,076,088 6, 2000 Paik et al. 6,078,914 6, 2000 Redfern U.S. PATENT DOCUMENTS 6,081,750 6, 2000 Hoffberg et al. 6,081,774 6, 2000 de Hita et al. 5,963,208 10, 1999 Dolan et al. 6,081,780 6, 2000 Lumelsky 5,963,924 10, 1999 Williams et al. 6,085.204 T/2000 Chijiwa et al. 5,963,964 10, 1999 Nielsen 6,088,671 T/2000 Gould et al. 5,966,126 10, 1999 Szabo 6,088,731 T/2000 Kiraly 5,970,446 10, 1999 Goldberg et al. 6,092,043 T/2000 Squires et al. 5,970.474 10, 1999 LeRoy et al. 6,094,649 T/2000 Bowen et al. 5,973,612 10, 1999 Deo et al. 6,097.391 8, 2000 Wilcox 5,973,676 10, 1999 Kawakura 6,101,468 8, 2000 Gould et al. 5,974,146 10, 1999 Randle et al. 6,101,470 8, 2000 Eide et al. 5,977,950 11, 1999 Rhyne 6,105,865 8, 2000 Hardesty 5,982,352 11, 1999 Pryor 6,108,627 8, 2000 Sabourin 5,982,891 11, 1999 Ginter et al. 6,108,640 8, 2000 Slotznick 5,982,902 11, 1999 Terano 6,111,562 8, 2000 Downs et al. 5,983, 179 11, 1999 Gould 6,111,572 8, 2000 Blair et al. 5,983,216 11, 1999 Kirsch et al. 6,115,686 9, 2000 Chung et al. 5,987,132 11, 1999 Rowney 6,116,907 9, 2000 Baker et al. 5,987,140 11, 1999 Rowney et al. 6,119,101 9, 2000 Peckover 5,987.401 11, 1999 Trudeau 6,121,960 9, 2000 Carroll et al. 5,987.404 11, 1999 Della Pietra et al. 6,122,340 9, 2000 Darley et al. 5,987.440 11, 1999 O'Neil et al. 6,122,614 9, 2000 Kahn et al. 5.990,887 11, 1999 Redpath et al. 6,122,616 9, 2000 Henton 5,991.441 11, 1999 Jourine 6,122,647 9, 2000 Horowitz et al. 5,995,460 11, 1999 Takagi et al. 6,125,284 9, 2000 Moore et al. 5,995,590 11, 1999 Brunet et al. 6,125,346 9, 2000 Nishimura et al. 5.998.972 12, 1999 Gong 6,125,356 9, 2000 Brockman et al. 5.999,169 12, 1999 Lee 6,129,582 10, 2000 Wilhite et al. 5.999,895 12, 1999 Forest 6,138,098 10, 2000 Shieber et al. 5.999,908 12, 1999 Abelow 6,138,158 10, 2000 Boyle et al. 5.999,927 12, 1999 Tukey et al. 6,141,642 10, 2000 Oh 6,006,274 12, 1999 Hawkins et al. 6,141,644 10, 2000 Kuhn et al. 6,009,237 12, 1999 Hirabayashi et al. 6,144,377 11, 2000 Oppermann et al. 6,011,585 1, 2000 Anderson 6,144,380 11, 2000 Shwarts et al. 6,014,428 1/2000 Wolf 6,144.938 11, 2000 Surace et al. 6,016,471 1, 2000 Kuhn et al. 6,144.939 11, 2000 Pearson et al. 6,018,705 1, 2000 Gaudet 6,151401 11, 2000 Annaratone 6,018,711 1, 2000 French-St. George et al. 6,154,551 11, 2000 Frenkel 6,020,881 2, 2000 Naughton et al. 6,154,720 11, 2000 Onishi et al. 6,023,536 2, 2000 Visser 6,157,935 12, 2000 Tran et al. 6,023,676 2, 2000 Erell 6,161,084 12, 2000 Messerly et al. 6,023,684 2, 2000 Pearson 6,161,087 12, 2000 Wightman et al. 6,024.288 2, 2000 Gottlich et al. 6,161.944 12, 2000 Leman 6,026,345 2, 2000 Shah et al. 6,163,769 12, 2000 Acero et al. 6,026,375 2, 2000 Hall et al. 6,163,809 12, 2000 Buckley 6,026,388 2, 2000 Liddy et al. 6,167,369 12, 2000 Schulze 6,026,393 2, 2000 Gupta et al. 6,169,538 1, 2001 Nowlan et al. 6,029,132 2, 2000 Kuhn et al. 6,172,948 1, 2001 Keller et al. 6,029, 135 2, 2000 Krasle 6,173,194 1, 2001 Vanttila 6,035,267 3, 2000 Watanabe et al. 6,173,251 1, 2001 Ito et al. 6,035.303 3, 2000 Baer et al. 6,173,261 1, 2001 Arai et al. 6,035,336 3, 2000 Lu et al. 6,173,263 1, 2001 Conkie 6,038,533 3, 2000 Buchsbaum et al. 6,173,279 1, 2001 Levin et al. 6,040,824 3, 2000 Maekawa et al. 6,177,905 1, 2001 Welch 6,041,023 3, 2000 Lakhansingh 6,177,931 1, 2001 Alexander et al. 6,047,255 4, 2000 Williamson 6,179,432 1, 2001 Zhang et al. 6,047.300 4, 2000 Walfish et al. 6,182,028 1, 2001 Karaali et al. 6,052,654 4, 2000 Gaudet et al. 6,185.533 2, 2001 Holm et al. 6,052,656 4, 2000 Suda et al. 6,188,391 2, 2001 Seely et al. 6,054,990 4, 2000 Tran 6,188,967 2, 2001 Kurtzberg et al. 6,055,514 4, 2000 Wren 6,188.999 2, 2001 Moody 6,055,531 4, 2000 Bennett et al. 6, 191939 2, 2001 Burnett 6,064,767 5, 2000 Muir et al. 6,192.253 2, 2001 Charlier et al. 6,064,959 5, 2000 Young et al. 6, 192,340 2, 2001 Abecassis 6,064,960 5, 2000 Bellegarda et al. 6, 195,641 2, 2001 Loring et al. 6,064.963 5, 2000 Gainsboro 6,205.456 3, 2001 Nakao 6,067,519 5, 2000 Lowry 6,208,044 3, 2001 Viswanadham et al. 6,069,648 5, 2000 Suso et al. 6,208.932 3, 2001 Ohmura et al. 6,070, 138 5, 2000 Iwata 6,208.956 3, 2001 Motoyama 6,070,139 5, 2000 Miyazawa et al. 6,208.964 3, 2001 Sabourin 6,070,140 5, 2000 Tran 6,208,967 3, 2001 Pauws et al. 6,070,147 5, 2000 Harms et al. 6,208,971 3, 2001 Bellegarda et al. 6,073,033 6, 2000 Campo 6.212,564 4, 2001 Harter et al. 6,073,036 6, 2000 Heikkinen et al. 6,216,102 4, 2001 Martino et al. 6,073,097 6, 2000 Gould et al. 6,216,131 4, 2001 Liu et al. 6,076,051 6, 2000 Messerly et al. 6,217, 183 4, 2001 Shipman 6,076,060 6, 2000 Lin et al. 6,222,347 4, 2001 Gong US 9,318.108 B2 Page 7

(56) References Cited 6,351,522 2, 2002 Vitikainen 6,351,762 2, 2002 Ludwig et al. U.S. PATENT DOCUMENTS 6,353,442 3, 2002 Masui 6,353,794 3, 2002 Davis et al. 6,226,403 5, 2001 Parthasarathy 6,356,287 3, 2002 Ruberry et al. 6,226,533 5, 2001 Akahane 6,356,854 3, 2002 Schubert et al. 6,226,614 5, 2001 Mizuno et al. 6,356,864 3, 2002 Foltz et al. 6,226,655 5, 2001 Borman et al. 6,356,905 3, 2002 Gershman et al. 6,230,322 5, 2001 Saib et al. 6,357,147 3, 2002 Darley et al. 6,232,539 5, 2001 Looney et al. 6,359,572 3, 2002 Vale 6,232,966 5, 2001 Kurlander 6,359,970 3, 2002 Burgess 6,233,545 5, 2001 Datig 6,360,227 3, 2002 Aggarwal et al. 6,233,547 5, 2001 Denber et al. 6,360,237 3, 2002 Schulz et al. 6,233.559 5, 2001 Balakrishnan 6,363.348 3, 2002 Besling et al. 6,233,578 5, 2001 Machihara et al. 6,366,883 4, 2002 Campbell et al. 6,237,025 5, 2001 Ludwig et al. 6,366,884 4, 2002 Bellegarda et al. 6,240,303 5, 2001 Katzur 6,374,217 4, 2002 Bellegarda 6,243,681 6, 2001 Guji et al. 6,377,530 4, 2002 Burrows 6,246,981 6, 2001 Papineni 6,377,925 4, 2002 Greene, Jr. et al. 6,248,946 6, 2001 Dwek 6,377,928 4, 2002 Saxena et al. 6,249,606 6, 2001 Kiraly et al. 6,381,593 4, 2002 Yano et al. 6,259,436 T/2001 Moon et al. 6,385,586 5/2002 Dietz 6,259,826 T/2001 Pollard et al. 6,385,662 5/2002 Moon et al. 6,260,011 T/2001 Heckerman et al. 6,389,114 5/2002 Dowens et al. 6,260,013 T/2001 Sejnoha 6,397, 183 5/2002 Baba et al. 6,260,016 T/2001 Holm et al. 6,397, 186 5/2002 Bush et al. 6,260,024 T/2001 Shkedy 6,400,806 6, 2002 Uppaluru 6,266,637 T/2001 Donovan et al. 6,401,065 6, 2002 Kanevsky et al. 6,268,859 T/2001 Andresen et al. 6.405,169 6, 2002 Kondo et al. 6,269,712 8, 2001 Zentmyer 6.405,238 6, 2002 Votipka 6,271,835 8, 2001 HoekSma 6,408,272 6, 2002 White et al. 6,272.456 8, 2001 De Campos 6,411,924 6, 2002 De Hita et al. 6,272.464 8, 2001 Kiraz et al. 6,411,932 6, 2002 Molnar et al. 6,275,795 8, 2001 Tzirkel-Hancock 6,415,250 T/2002 Van Den Akker 6,275,824 8, 2001 O'Flaherty et al. 6,417,873 T/2002 Fletcher et al. 6,278.443 8, 2001 Amro et al. 6.421,305 T/2002 Gioscia et al. 6,278,970 8, 2001 Milner 6.421,672 T 2002 McAllister et al. 6,282,507 8, 2001 Horiguchi et al. 6.421,707 T/2002 Miller et al. 6,285,785 9, 2001 Bellegarda et al. 6,424.944 T/2002 Hikawa 6,285,786 9, 2001 Seni et al. 6,430,551 8, 2002 Thelen et al. 6,289,085 9, 2001 Miyashita et al. 6,434,522 8, 2002 Tsuboka 6,289,124 9, 2001 Okamoto 6,434,524 8, 2002 Weber 6,289,301 9, 2001 Higginbotham et al. 6,434,604 8, 2002 Harada et al. 6,289.353 9, 2001 Hazlehurst et al. 6,437,818 8, 2002 Ludwig et al. 6,292,772 9, 2001 Kantrowitz 6,438,523 8, 2002 Oberteuffer et al. 6,292,778 9, 2001 Sukkar 6,442,518 8, 2002 Van Thong et al. 6,295,390 9, 2001 Kobayashi et al. 6,442,523 8, 2002 Siegel 6,295,541 9, 2001 Bodnar et al. 6,446,076 9, 2002 Burkey et al. 6,297.818 10, 2001 Ulrich et al. 6,448,485 9, 2002 Barile 6,298,314 10, 2001 Blackadar et al. 6,448,986 9, 2002 Smith 6,298,321 10, 2001 Karlov et al. 6,449,620 9, 2002 Draper et al. 6,300,947 10, 2001 Kanevsky 6,453,281 9, 2002 Walters et al. 6,304,844 10, 2001 Pan et al. 6,453,292 9, 2002 Ramaswamy et al. 6,304,846 10, 2001 George et al. 6,453,315 9, 2002 Weissman et al. 6,307,548 10, 2001 Flinchem et al. 6,456,616 9, 2002 Rantanen 6,308, 149 10, 2001 Gaussier et al. 6,456.972 9, 2002 Gladstein et al. 6,310,610 10, 2001 Beaton et al. 6,460,015 10, 2002 Hetherington et al. 6,311,157 10, 2001 Strong 6,460,029 10, 2002 Fries et al. 6,311, 189 10, 2001 deVries et al. 6,462,778 10, 2002 Abram et al. 6,317,237 11, 2001 Nakao et al. 6,463,128 10, 2002 Elwin 6,317,594 11, 2001 Gossman et al. 6,466,654 10, 2002 Cooper 6,317,707 11, 2001 Bangalore et al. 6.467,924 10, 2002 Shipman 6,317,831 11, 2001 King 6,469,712 10, 2002 Hilpert, Jr. et al. 6,321,092 11, 2001 Fitch et al. 6,469,722 10, 2002 Kinoe et al. 6,321, 179 11, 2001 Glance et al. 6,469,732 10, 2002 Chang et al. 6,323,846 11, 2001 Westerman et al. 6,470,347 10, 2002 Gillam 6,324,502 11, 2001 Handel et al. 6,473,630 10, 2002 Baranowski et al. 6,324,512 11, 2001 Junqua et al. 6,477,488 11, 2002 Bellegarda 6,330,538 12, 2001 Breen 6,477,494 11, 2002 Hyde-Thomson et al. 6,331,867 12, 2001 Eberhard et al. 6,487,533 11, 2002 Hyde-Thomson et al. 6,332,175 12, 2001 Birrell et al. 6,487,534 11, 2002 Thelen et al. 6,334,103 12, 2001 Surace et al. 6,487,663 11, 2002 Jaisimha et al. 6,335,722 1, 2002 Tani et al. 6,489.951 12, 2002 Wong et al. 6,336,365 1, 2002 Blackadar et al. 6,490,560 12, 2002 Ramaswamy et al. 6,336,727 1, 2002 Kim 6,493,006 12, 2002 Gourdol et al. 6,340,937 1, 2002 Stepita-Klauco 6,493,428 12, 2002 Hillier 6,341,316 1, 2002 Kloba et al. 6,493,652 12, 2002 Ohlenbusch et al. 6,343,267 1, 2002 Kuhn et al. 6,493,667 12, 2002 De Souza et al. 6,345,250 2, 2002 Martin 6,499,013 12, 2002 Weber US 9,318.108 B2 Page 8

(56) References Cited 6,623.529 9, 2003 Lakritz 6,625,583 9, 2003 Silverman et al. U.S. PATENT DOCUMENTS 6,628,808 9, 2003 Bach et al. 6,631, 186 10, 2003 Adams et al. 6,499,014 12, 2002 Chihara 6,631,346 10, 2003 Karaorman et al. 6,499,016 12, 2002 Anderson et al. 6,633,741 10, 2003 Posa et al. 6,501,937 12, 2002 Ho et al. 6,633,846 10, 2003 Bennett et al. 6,502,194 12, 2002 Berman et al. 6,633,932 10, 2003 Bork et al. 6,505,158 1, 2003 Conkie 6,642.940 11/2003 Dakss et al. 6,505, 175 1, 2003 Silverman et al. 6,643,401 11/2003 Kashioka et al. 6,505, 183 1, 2003 Loofbourrow et al. 6,643,824 11/2003 Bates et al. 6,507,829 1, 2003 Richards et al. 6,647,260 11/2003 Dusse et al. 6,510,406 1, 2003 Marchisio 6,650,735 11/2003 Burton et al. 6,510,417 1, 2003 Woods et al. 6,651,042 11/2003 Field et al. 6,513,008 1, 2003 Pearson et al. 6,651,218 11/2003 Adler et al. 6,513,063 1, 2003 Julia et al. 6,654,740 11/2003 Tokuda et al. 6,519,565 2, 2003 Clements et al. 6,658,389 12, 2003 Alpdemir 6,519,566 2, 2003 Boyer et al. 6,658.408 12, 2003 Yano et al. 6,523,026 2, 2003 Gillis 6,658,577 12, 2003 Huppi et al. 6,523,061 2, 2003 Halverson et al. 6,662,023 12, 2003 Helle 6,523,172 2, 2003 Martinez-Guerra et al. 6,665,639 12, 2003 Mozer et al. 6,526,351 2, 2003 Whitham 6,665,640 12, 2003 Bennett et al. 6,526,382 2, 2003 Yuschik 6,665,641 12, 2003 Coorman et al. 6,526,395 2, 2003 Morris 6,671,672 12, 2003 Heck 6,529,592 3, 2003 Khan 6,671.683 12, 2003 Kanno 6,529,608 3, 2003 Gersabeck et al. 6,671,856 12, 2003 Gillam 6,532.444 3, 2003 Weber 6,675,169 1, 2004 Bennett et al. 6,532,446 3, 2003 King 6,675,233 1, 2004 Du et al. 6,535,610 3, 2003 Stewart 6,677,932 1, 2004 Westerman 6,535,852 3, 2003 Eide 6,680,675 1, 2004 Suzuki 6,535,983 3, 2003 McCormack et al. 6,684,187 1, 2004 Conkie 6,536,139 3, 2003 Darley et al. 6,684,376 1, 2004 Kerzman et al. 6,538,665 3, 2003 Crow et al. 6,690,387 2, 2004 Zimmerman et al. 6,542,171 4, 2003 Satou et al. 6,690,800 2, 2004 Resnick 6,542,584 4, 2003 Sherwood et al. 6,690,828 2, 2004 Meyers 6,546,262 4, 2003 Freadman 6,691,064 2, 2004 Vroman 6,546,367 4, 2003 Otsuka 6,691,090 22004 Laurila et al. 6,546,388 4, 2003 Edlund et al. 6,691,111 2, 2004 Lazaridis et al. 6,549,497 4, 2003 Miyamoto et al. 6,691, 151 2, 2004 Cheyer et al. 6,553,343 4, 2003 Kagoshima et al. 6,694,295 2, 2004 Lindholm et al. 6,553,344 4, 2003 Bellegarda et al. 6,694,297 2, 2004 Sato 6,556,971 4, 2003 Rigsby et al. 6,697,780 2, 2004 Beutnagel et al. 6,556,983 4, 2003 Altschuler et al. 6,697,824 2, 2004 Bowman-Amuah 6,560,903 5/2003 Darley 6,701,294 3, 2004 Ball et al. 6,563,769 5/2003 Van Der Meulen 6,701,305 3, 2004 Holt et al. 6,564, 186 5/2003 Kiraly et al. 6,701,318 3, 2004 Fox et al. 6,567,549 5/2003 Marianetti et al. 6,704,015 3, 2004 Bovarnicket al. 6,570,557 5/2003 Westerman et al. 6,704,034 3, 2004 Rodriguez et al. 6,570,596 5/2003 Frederiksen 6,704,698 3, 2004 Paulsen, Jr. et al. 6,582,342 6, 2003 Kaufman 6,704,710 3, 2004 Strong 6,583,806 6, 2003 Ludwig et al. 6,708,153 3, 2004 Brittan et al. 6,584,464 6, 2003 Warthen 6,711,585 3, 2004 Copperman et al. 6,587,403 T/2003 Keller et al. 6,714,221 3, 2004 Christie et al. 6,587.404 T/2003 Keller et al. 6,716, 139 4, 2004 Hosseinzadeh-Dolkhani et al. 6,590,303 T/2003 Austin et al. 6,718,324 4, 2004 Edlund et al. 6,591,379 T/2003 LeVine et al. 6,718,331 4, 2004 Davis et al. 6,594,673 T/2003 Smith et al. 6,720,980 4, 2004 Lui et al. 6,594,688 T/2003 Ludwig et al. 6,721,728 4, 2004 McGreevy 6,597,345 T/2003 Hirshberg 6,721,734 4, 2004 Subasic et al. 6,598,021 T/2003 Shambaugh et al. 6,724,370 4, 2004 Dutta et al. 6,598,022 T/2003 Yuschik 6,725, 197 4, 2004 Wuppermann et al. 6,598,039 T/2003 Livowsky 6,728,675 4, 2004 Maddalozzo, Jr. et al. 6,598,054 T/2003 Schuetze et al. 6,728,681 4, 2004 Whitham 6,601,026 T/2003 Appelt et al. 6,728,729 4, 2004 Jawa et al. 6,601.234 T/2003 Bowman-Amuah 6,731,312 5, 2004 Robbin 6,603,837 8, 2003 Kesanupalli et al. 6,732,142 5, 2004 Bates et al. 6,604,059 8, 2003 Strubbe et al. 6,735,632 5, 2004 Kiraly et al. 6,606,101 8, 2003 Malamud et al. 6,738,738 5, 2004 Henton 6,606,388 8, 2003 Townsend et al. 6,741,264 5, 2004 Lesser 6,606,632 8, 2003 Saulpaugh et al. 6,742,021 5, 2004 Halverson et al. 6,611,789 8, 2003 Darley 6,751,592 6, 2004 Shiga 6,615,172 9, 2003 Bennett et al. 6,751,595 6, 2004 Busayapongchai et al. 6,615, 175 9, 2003 Gazdzinski 6,751,621 6, 2004 Calistri-Yeh et al. 6,615, 176 9, 2003 Lewis et al. 6,754,504 6, 2004 Reed 6,615,220 9, 2003 Austin et al. 6,757,362 6, 2004 Cooper et al. 6,621,768 9, 2003 Keller et al. 6,757,365 6, 2004 Bogard 6,621,892 9, 2003 Banister et al. 6,757,646 6, 2004 Marchisio 6,622,121 9, 2003 Crepy et al. 6,757,653 6, 2004 Buth et al. 6,622, 136 9, 2003 Russell 6,757,718 6, 2004 Halverson et al. US 9,318.108 B2 Page 9

(56) References Cited 6,882,747 B2 4/2005 Thawonmas et al. 6,882.955 B1 4/2005 Ohlenbusch et al. U.S. PATENT DOCUMENTS 6,882,971 B2 4/2005 Craner 6,885,734 B1 4/2005 Eberle et al. 6,760,412 B1 7/2004 Loucks 6,889,361 B1 5/2005 Bates et al. 6,760,700 B2 7/2004 Lewis et al. 6,895,084 B1 5/2005 Saylor et al. 6,760,754 B1 7/2004 Isaacs et al. 6,895,257 B2 5/2005 Boman et al. 6,762,741 B2 7/2004 Weindorf 6,895.380 B2 5/2005 Sepe, Jr. 6,762,777 B2 7/2004 Carroll 6,895,558 B1 5/2005 Loveland 6,763,089 B2 7/2004 Feigenbaum 6,898,550 B1 5/2005 Blackadar et al. 6,766,294 B2 7/2004 MacGinite et al. 6,901,364 B2 5/2005 Nguyen et al. 6,766,320 B1 7/2004 Wang et al. 6,901,399 B1 5/2005 Corston et al. 6,766,324 B2 7/2004 Carlson et al. 6,904,405 B2 6/2005 Suominen 6,768,979 B1 7/2004 Menendez-Pidal et al. 6,907, 112 B1 6/2005 Guedalia et al. 6,772,123 B2 8/2004 Cooklev et al. 6,907,140 B2 6/2005 Matsugu et al. 6,772,195 B1 8, 2004 Hatlelid et al. 6,910,004 B2 6/2005 Tarbouriech et al. 6,772,394 B1 8/2004 Kamada 6,910,007 B2 6/2005 Stylianou et al. 6,775,358 B1 8/2004 Breitenbach et al. 6,910, 186 B2 6/2005 Kim 6,778,951 B1 8/2004 Contractor 6,911,971 B2 6/2005 Suzuki et al. 6,778,952 B2 8/2004 Bellegarda 6,912,407 B1 6/2005 Clarke et al. 6,778,962 B1 8/2004 Kasai et al. 6,912,498 B2 6/2005 Stevens et al. 6778970 B2 8, 2004 Au 6,912,499 B1 6/2005 Sabourin et al. 6,778,979 B2 8/2004 Grefenstette et al. 6,915,138 B2 7/2005 Kraft 6,782,510 B1 8, 2004 Gross et al. 6,915,246 B2 7/2005 Gusler et al. 6,784,901 B1 8/2004 Harvey et al. 6,917,373 B2 7/2005 Vong et al. 6,789,094 B2 9/2004 Rudoffetal. 6,918,677 B2 7/2005 Shipman 6,789.231 B1 9/2004 Reynar et al. 6,924,828 B1 8/2005 Hirsch 6,790,704 B2 9/2004 Doyle et al. 6,925.438 B2 8, 2005 Mohamed et al. 6,792,082 B1 9/2004 Levine 6,928,149 B1 8/2005 Panjwani et al. 6.792,086 B1 9/2004 Saylor et al. 6928,614 B1 8/2005 Everhart 6,792.407 B2 9/2004 Kibre et al. 6,931,255 B2 8/2005 Mekuria 6,794,566 B2 9, 2004 Pachet 6,931,384 B1 8, 2005 Horvitz et al. 6,795,059 B2 9, 2004 Endo 6,932,708 B2 8, 2005 Yamashita et al. 6,799,226 B1 9/2004 Robbin et al. 6,934,394 B1 8/2005 Anderson 6,801,604 B2 10/2004 Maes et al. 6,934,684 B2 8/2005 Alpdemir et al. 6,801,964 B1 10/2004 Mahdavi 6,934,756 B2 8/2005 Maes. 6,803,905 B1 10/2004 Capps et al. 6,934,812 B1 8, 2005 Robbin et al. 6,804,649 B2 10/2004 Miranda 6,937,975 B1 8/2005 Elworthy 6,804,677 B2 10/2004 Shadmon et al. 6,937,986 B2 8/2005 Denenberg et al. 6,807,536 B2 10/2004 Achlioptaset al. 6,944,593 B2 9/2005 KuZunuki et al. 6,807,574 B1 10/2004 Partoviet al. 6,948,094 B2 9, 2005 Schultz et al. 6,810,379 B1 10/2004 Vermeulen et al. 6,950,087 B2 9/2005 Knox et al. 6,813,218 B1 1 1/2004 Antonelli et al. 6,950,502 B1 9/2005 Jenkins 6,813,491 B1 1 1/2004 McKinney 6.952,799 B2 10/2005 Edwards et al. 6,813,607 B1 1 1/2004 Faruquie et al. 6,954,755 B2 10/2005 Reisman 6,816,578 B1 1 1/2004 Kredo et al. 6,954,899 B1 10/2005 Anderson 6,820,055 B2 11/2004 Saindon et al. 6.956,845 B2 10/2005 Baker et al. 6,829,018 B2 12/2004 Lin et al. 6.957,076 B2 10/2005 Hunzinger 6,829,603 B1 12/2004 Cahi et al. 6.957,183 B2 10/2005 Malayath et al. 6,832,194 B1 12/2004 Mozer et al. 6,960,734 B1 1 1/2005 Park 6,832,381 B1 12/2004 Mathur et al. 6.961,699 B1 1 1/2005 Kahn et al. 6,836,651 B2 12/2004 Segal et al. 6,961,912 B2 11/2005 Aoki et al. 6,836,760 B1 12/2004 Bellegarda et al. 6,963,841 B2 11/2005 Handal et al. 6,839,464 B2 1/2005 Hawkins et al. 6,964,023 B2 11/2005 Maes et al. 6,839,669 B1 1/2005 Gould et al. 6,965,376 B2 11/2005 Tani et al. 6,839,670 B1 1/2005 Stammler et al. 6,965,863 B1 * 1 1/2005 Zuberec ...... G1 OL15, 22 6,839,742 B1 1/2005 Dyer et al. TO4,231 6,842,767 B1 1/2005 Partoviet al. 6,968,311 B2 11/2005 Knockeart et al. 6,847,966 B1 1/2005 Sommer et al. 6,970,820 B2 11/2005 Junqua et al. 6,847,979 B2 1/2005 Allemang et al. 6,970,881. B1 11/2005 Mohan et al. 6,850,775 B1 2/2005 Berg 6,970,915 B1 1 1/2005 Partoviet al. 6.850,887 B2 2/2005 Epstein et al. 6,970,935 B1 1 1/2005 Maes 6,851,115 B1 2/2005 Cheyer et al. 6,976,090 B2 12/2005 Ben-Shaul et al. 6,856,259 B1 2/2005 Sharp 6,978,127 B1 12/2005 Bulthuis et al. 6,857,800 B2 2/2005 Zhang et al. 6,978,239 B2 12/2005 Chu et al. 6,859,931 B1 2/2005 Cheyer et al. 6.980.949 B2 12/2005 Ford 6,862.568 B2 3/2005 Case 6,980,955 B2 12/2005 Okutani et al. 6,862,710 B1 3/2005 Marchisio 6,983,251 B1 1/2006 Umemoto et al. 6,865,533 B2 3/2005 Addison et al. 6,985,858 B2 1/2006 Frey et al. 6,868,045 B1 3/2005 Schroder 6,985,865 B1 1/2006 Packingham et al. 6,868,385 B1 3/2005 Gerson 6,988,071 B1 1/2006 GaZdzinski 6,870,529 B1 3/2005 Davis 6,990,450 B2 1/2006 Case et al. 6,871,346 B1 3/2005 Kumbalimutt et al. 6,996,520 B2 2/2006 Levin 6,873,986 B2 3/2005 McConnell et al. 6,996,531 B2 2/2006 Korall et al. 6,876.947 B1 4/2005 Darley et al. 6,996,575 B2 2/2006 Cox et al. 6,877,003 B2 4/2005 Ho et al. 6,999,066 B2 2/2006 Litwiller 6,879,957 B1 4/2005 Pechter et al. 6,999,914 B1 2/2006 Boerner et al. 6,882,335 B2 4/2005 Saarinen 6,999,925 B2 2/2006 Fischer et al. 6,882.337 B2 4/2005 Shetter 6,999,927 B2 2/2006 Mozer et al. US 9,318.108 B2 Page 10

(56) References Cited 7,111,248 9, 2006 Mulvey et al. 7,111,774 9, 2006 Song U.S. PATENT DOCUMENTS 7,113,803 9, 2006 Dehlin 7,113,943 9, 2006 Bradford et al. 7,000, 189 2, 2006 Dutta et al. 7,115,035 10, 2006 Tanaka 7,002.556 2, 2006 Tsukada et al. 7,117,231 10, 2006 Fischer et al. 7,003,099 2, 2006 Zhang et al. 7,123,696 10, 2006 Lowe 7,003.463 2, 2006 Maes et al. 7,124,081 10, 2006 Bellegarda 7,003,522 2, 2006 Reynar et al. 7,124,082 10, 2006 Freedman 7,006,969 2, 2006 Atal 7,124,164 10, 2006 Chemtob 7,007,239 2, 2006 Hawkins et al. 7,127,046 10, 2006 Smith et al. 7,010,581 3, 2006 Brown et al. 7,127,396 10, 2006 Chu et al. 7,013,289 3, 2006 Horn et al. 7,127.403 10, 2006 Saylor et al. 7,013,308 3, 2006 Tunstall-Pedoe 7,133,900 11, 2006 Szeto 7,013,429 3, 2006 Fujimoto et al. 7,136,710 11, 2006 Hoffberg et al. 7,015,894 3, 2006 Morohoshi 7,136,818 11, 2006 Cosatto et al. 7,020,685 3, 2006 Chen et al. 7,137,126 11, 2006 Coffman ...... GO6F 17,30899 7,024,363 4, 2006 Comerford et al. TO4/200 7,024,364 4, 2006 Guerra et al. 7,139,697 B2 11, 2006 Häkkinen et al. 7,024,366 4, 2006 Deyoe et al. 7,139,714 B2 11, 2006 Bennett et al. 7,024,460 4, 2006 Koopmas et al. 7,139,722 B2 11, 2006 Perrella et al. 7,027,568 4, 2006 Simpson et al. 7,143,028 B2 11, 2006 Hillis et al. 7,027,974 4, 2006 Busch et al. 7,143,038 B2 11, 2006 Katae 7,027,990 4, 2006 Sussman 7,143,040 B2 11, 2006 Durston et al. 7,028,252 4, 2006 Baru et al. 7,146,319 B2 12, 2006 Hunt 7,030,861 4, 2006 Westerman et al. 7,146437 B2 12, 2006 Robbin et al. 7,031,530 4, 2006 Driggs et al. 7,149,319 B2 12, 2006 Roeck 7,031,909 4, 2006 Mao et al. 7,149,695 B1 12, 2006 Bellegarda 7,035,794 4, 2006 Sirivara 7,149,964 B1 12, 2006 Cottrille et al. 7,035,801 4, 2006 Jimenez-Feltstrom 7,152,070 B1 12, 2006 Musick et al. 7,035,807 4, 2006 Brittain et al. 7,152,093 B2 12, 2006 Ludwig et al. 7,036,128 4, 2006 Julia et al. 7,154,526 B2 12, 2006 Foote et al. 7,038,659 5, 2006 Rajkowski 7,155,668 B2 12, 2006 Holland et al. 7,039,588 5, 2006 Okutani et al. 7,158,647 B2 1/2007 AZima et al. 7,043,420 5, 2006 Ratnaparkhi 7,159,174 B2 1/2007 Johnson et al. 7,043,422 5, 2006 Gao et al. 7,162,412 B2 1/2007 Yamada et al. 7,046,230 5, 2006 Zadesky et al. 7,162,482 B1 1/2007 Dunning 7,046,850 5, 2006 Braspenning et al. 7,165,073 B2 1/2007 Vandersluis 7,047, 193 5, 2006 Bellegarda 7,166,791 B2 1/2007 Robbin et al. 7,050,550 5, 2006 Steinbiss et al. 7,171,360 B2 1/2007 Huang et al. 7,050,976 5, 2006 Packingham 7,174,042 B1 2, 2007 Simmons et al. 7,050,977 5, 2006 Bennett 7,174,295 B1 2, 2007 Kivimaki 7,051,096 5, 2006 Krawiec et al. 7,174,297 B2 2, 2007 Guerra et al. 7,054,419 5, 2006 Culliss 7,174,298 B2 2, 2007 Sharma 7,054,888 5, 2006 Lachapelle et al. 7, 177,794 B2 2, 2007 Mani et al. 7.057,607 6, 2006 Mayoraz et al. 7, 177,798 B2 2, 2007 Hsu et al. 7,058.569 6, 2006 Coorman et al. 7, 177,817 B1 2, 2007 Khosla et al. 7,058,888 6, 2006 Gjerstad et al. 7,181,386 B2 2, 2007 Mohri et al. 7,058,889 6, 2006 Trovato et al. 7,181,388 B2 2, 2007 Tian 7,062,223 6, 2006 Gerber et al. 7,184,064 B2 2, 2007 Zimmerman et al. 7,062,225 6, 2006 White 7,185,276 B2 2, 2007 Keswa 7,062,428 6, 2006 Hogenhout et al. 7,188,085 B2 3, 2007 Pelletier 7,062,438 6, 2006 Kobayashi et al. 7, 190,351 B1 3, 2007 Goren 7,065,185 6, 2006 Koch 7, 190,794 B2 3, 2007 Hinde 7,065.485 6, 2006 Chong-White et al. 7,191,118 B2 3, 2007 Bellegarda 7,069,213 6, 2006 Thompson 7,191,131 B1 3, 2007 Nagao 7,069,220 6, 2006 Coffman et al. 7,193,615 B2 3, 2007 Kim et al. 7,069,560 6, 2006 Cheyer et al. 7,194, 186 B1 3, 2007 Strub et al. 7,072,686 T/2006 Schrager 7,194,413 B2 3, 2007 Mahoney et al. 7,072,941 T/2006 Griffin et al. 7,194,471 B1 3, 2007 Nagatsuka et al. 7,076,527 T/2006 Bellegarda et al. 7,194,611 B2 3, 2007 Bear et al. 7,079,713 T/2006 Simmons 7,194,699 B2 3, 2007 Thomson et al. 7,082,322 T/2006 Harano 7,197,120 B2 3, 2007 Luehrig et al. 7,084,758 8, 2006 Cole 7, 197460 B1 3, 2007 Gupta 7,084,856 8, 2006 Huppi 7,200,550 B2 4, 2007 Menezes et al. 7,085,723 8, 2006 Ross et al. 7,200,558 B2 4, 2007 Kato et al. 7,085,960 8, 2006 Bouat et al. 7,200,559 B2 4, 2007 Wang 7,088,345 8, 2006 Robinson et al. 7,203,297 B2 4, 2007 Vitikainen ...... HO4M 3/533 7,089,292 8, 2006 Rodericket al. 379,207.O2 7,092.370 8, 2006 Jiang et al. 7,203,646 B2 4, 2007 Bennett 7,092,887 8, 2006 Mozer et al. 7,206.809 B2 4, 2007 Ludwig et al. 7,092,928 8, 2006 Elad et al. 7,216,008 B2 5/2007 Sakata 7,092,950 8, 2006 Wong et al. 7,216,073 B2 5/2007 Lavi et al. 7,093,.693 8, 2006 Gazdzinski 7,216,080 B2 5/2007 Tsiao 7,095,733 8, 2006 Yarlagadda et al. 7,218,920 B2 5/2007 Hyon 7,096, 183 8, 2006 Junqua 7,218,943 B2 5/2007 Klassen et al. 7,100,117 8, 2006 Chwa et al. 7.219,063 B2 5/2007 Schalk et al. 7,103,548 9, 2006 Squibbs et al. 7,219,123 B1 5/2007 Fiechter et al. 7,107,204 9, 2006 Liu et al. 7,225,125 B2 5/2007 Bennett et al. US 9,318.108 B2 Page 11

(56) References Cited 7,383,170 B2 6/2008 Mills et al. 7,386,438 B1 6/2008 Franz et al. U.S. PATENT DOCUMENTS 7,386,449 B2 6/2008 Sun et al. 7,386,799 B1 6/2008 Clanton et al. 7.228,278 B2 6/2007 Nguyen et al. 7,389,224 B1 6/2008 Elworthy 7.231.343 B1 6/2007 Treadgold et al. 7,389,225 B1 6/2008 Jensen et al. 7,231,597 B1 6/2007 Braun et al. 7,392,185 B2 62008 Bennett 7,233,790 B2 6/2007 Kjellberg et al. 7,394,947 B2 7/2008 Liet al. 7,233,904 B2 6, 2007 Luisi 7,398,209 B2 T/2008 Kennewick 7,234,026 B2 6/2007 Robbin et al. 7401,300 B2 7/2008 Nurmi 7,236,932 B1 6/2007 Grajski 7,403,938 B2 7/2008 Harrison et al. 7,240,002 B2 7/2007 Minamino et al. 7,403,941 B2 7/2008 Bedworth et al. 7,243,130 B2 7/2007 Horvitz et al. 7,404,143 B2 7/2008 Freelander et al. 7,243,305 B2 7/2007 Schabes et al. 7,409,337 B1 8/2008 Potter et al. 7,246,118 B2 7/2007 Chastain et al. 7,409,347 B1 8/2008 Bellegarda 7,246,151 B2 7/2007 Isaacs et al. 7,412,389 B2 8/2008 Yang 7,248,900 B2 7/2007 Deeds et al. 7,412,470 B2 8/2008 Masuno et al. 7,251,454 B2 T/2007 White 7,415, 100 B2 8/2008 Cooper et al. 7,254,773 B2 8, 2007 Bates et al. 7,418,389 B2 8, 2008 Chu et al. 7,257,537 B2 8, 2007 Ross et al. 7,418,392 B1 8, 2008 MoZer et al. 7,259,752 B1 8, 2007 Simmons 7,426,467 B2 9, 2008 Nashida et al. 7,260,529 B1 8/2007 Lengen 7,426,468 B2 9, 2008 Coifman et al. 7,263,373 B2 8, 2007 Mattisson 7.427.024 B1 9, 2008 Gazdzinski et al. 7,266,189 B1 9/2007 Day 7,428,541 B2 9, 2008 Houle 7,266.495 B1 9/2007 Beaufays et al. 7,433,869 B2 10/2008 Gollapudi 7,266,496 B2 9/2007 Wang et al. 7,433,921 B2 10/2008 Ludwig et al. 7,266.499 B2 9, 2007 Surace et al. 7,441,184 B2 10/2008 Frerebeau et al. 7,269,544 B2 9/2007 Simske 7,443,316 B2 10/2008 Lim 7,269,556 B2 9/2007 Kiss et al. 7,444,589 B2 10/2008 Zellner 7,272,224 B1 9/2007 Normile et al. 7.447,360 B2 11/2008 Lietal. 7,275,063 B2 9, 2007 Horn 7.447,635 B1 1 1/2008 Konopka et al. 7,277,088 B2 10/2007 Robinson et al. 7.447,637 B1 * 1 1/2008 Grant ...... G1OL 15, 193 7,277,854 B2 10/2007 Bennett et al. TO4,231 7,277,855 B1 10/2007 Acker et al. 7,451,081 B1 1 1/2008 Gajic et al. 7,280,958 B2 10/2007 Pavlov et al. 7,454,351 B2 11/2008 Jeschke et al. 7,283,072 B1 10/2007 Plachta et al. 7.460,652 B2 12/2008 Chang 7,289,102 B2 10/2007 Hinckley et al. 7461,043 B2 122008 Hess 7,290,039 B1 10/2007 Lisitsa et al. 7.467,087 B1 12/2008 Gillicket al. 7,292,579 B2 11/2007 Morris 7,467,164 B2 12/2008 Marsh 7,292,979 B2 11/2007 Karas et al. 7,472,061 B1 12/2008 Alewine et al. 7,296,230 B2 11/2007 Fukatsu et al. 7,472,065 B2 12/2008 Aaron et al. 7,299.033 B2 11/2007 Kjellberg et al. 7,475,010 B2 1/2009 Chao 7,302,392 B1 1 1/2007 Thenthiruperaietal. 7,475,063 B2 1/2009 Datta et al. 7,302,394 B1 * 1 1/2007 Baray et al...... 704/257 7,477,238 B2 1/2009 Fux et al. 7,302,686 B2 11/2007 Togawa 7,477,240 B2 1/2009 Yanagisawa 7,308.404 B2 12/2007 Venkataraman et al. 7,478,037 B2 1/2009 Strong 7,308.408 B1 12/2007 Stifelman et al. 7,478,091 B2 1/2009 Mojsilovic et al. 7,310,329 B2 12/2007 Vieri et al. 7,478,129 B1 1/2009 Chemtob 7,310,600 B1 12/2007 Garner et al. 7,479,948 B2 1/2009 Kim et al. 7,310,605 B2 12/2007 Janakiraman et al. 7,479,949 B2 1/2009 Jobs et al. 7.313,523 B1 12/2007 Bellegarda et al. 7,483,832 B2 1/2009 Tischer 7,315,809 B2 1/2008 Xun 7,483,894 B2 1, 2009 Cao 7.315,818 B2 1/2008 Stevens et al. 7.487,089 B2 2/2009 Mozer 7,319,957 B2 1/2008 Robinson et al. 7487,093 B2 2/2009 Mutsuno et al. 7,321,783 B2 1/2008 Kim 7,490,034 B2 2/2009 Finnigan et al. 7,322,023 B2 1/2008 Shulman et al. 7,490,039 B1 2/2009 Shaffer et al. 7,324,833 B2 1/2008 White et al. 7,493,560 B1 2/2009 Kipnes et al. 7.324,947 B2 * 1/2008 Jordan ...... HO4N 21,4782 7,496.498 B2 2/2009 Chu et al. 704/257 7,496.512 B2 2/2009 Zhao et al. 7,328, 155 B2 2/2008 Endo et al. 7,499,923 B2 3/2009 Kawatani 7,345,670 B2 3/2008 Armstrong 7,502,738 B2 3/2009 Kennewicket al. 7,345,671 B2 3/2008 Robbin et al. 7,505,795 B1 3/2009 Lim et al. 7,349.953 B2 3/2008 Lisitsa et al. 7,508.324 B2 3/2009 Suraqui 7,353,139 B1 4/2008 Burrell et al. 7,508,373 B2 3/2009 Lin et al. 7,359,493 B1 4/2008 Wang et al. 7,516,123 B2 4/2009 Betz et al. 7.359,671 B2 4/2008 Richenstein et al. 7,519,327 B2 4/2009 White 7,359,851 B2 4/2008 Tong et al. 7,522,927 B2 4/2009 Fitch et al. 7,360,158 B1 4/2008 Beeman 7,523,036 B2 4/2009 Akabane et al. 7.362,738 B2 4/2008 Taube et al. 7,523,108 B2 4/2009 Cao 7.363.227 B2 4/2008 Mapes-Riordan et al. 7,526,466 B2 42009 Au 7,365,260 B2 4/2008 Kawashima 7,526,738 B2 4/2009 Ording et al. 7.366,461 B1 4/2008 Brown 7,528,713 B2 5/2009 Singh et al. 7,373,612 B2 5/2008 Risch et al. 7,529,671 B2 5/2009 Rockenbeck et al. 7,376,556 B2 5/2008 Bennett 7,529,676 B2 5/2009 Koyama 7,376,632 B1 5/2008 Sadek et al. 7,535,997 B1 5/2009 McQuaide, Jr. et al. 7,376,645 B2 5/2008 Bernard 7,536,029 B2 5/2009 Choi et al. 7.378.963 B1 5/2008 Begault et al. 7,536,565 B2 5/2009 Girish et al. 7,379,874 B2 5/2008 Schmid et al. 7,538,685 B1 5/2009 Cooper et al. 7,380.203 B2 5/2008 Keely et al. 7,539,619 B1 5/2009 Seligman et al. US 9,318.108 B2 Page 12

(56) References Cited 7,681,126 B2 3/2010 Roose 7,683,886 B2 3/2010 Willey U.S. PATENT DOCUMENTS 7,683,893 B2 3/2010 Kim 7,684,985 B2 3/2010 Dominach et al. 7,539,656 B2 5/2009 Fratkina et al. 7,684,990 B2 3/2010 Caskey et al. 7,541,940 B2 6/2009 Upton 7,684.991 B2 3/2010 Stohr et al. 7,542,967 B2 6/2009 Hurst-Hiller et al. 7,689,245 B2 3/2010 Cox et al. 7,543,232 B2 6/2009 Easton et al. 7,689408 B2 3/2010 Chen et al. 7.546,382 B2 6/2009 Healey et al. 7,689409 B2 3/2010 Heinecke 7.546,529 B2 6/2009 Reynar et al. 7,689,421 B2 3/2010 Lietal. 7,548,895 B2 6/2009 Pulsipher 7,693,715 B2 4/2010 Hwang et al. 7.552,045 B2 6/2009 Barliga et al. 7,693,717 B2 4/2010 Kahn et al. 7.552,055 B2 * 6/2009 Lecoeuche ...... TO4/270.1 7,693,719 B2 4/2010 Chu et al. 7.555.431 B2 6, 2009 Bennett 7,693,720 B2 4/2010 Kennewick 7.555.496 B1 6/2009 Lantrip et al. 7,698,131 B2 4/2010 Bennett 7,558,381 B1 7/2009 Ali et al. 7,702,500 B2 4/2010 Blaedow 7,558,730 B2 7/2009 Davis et al. 7,702,508 B2 4/2010 Bennett 7,559,026 B2 7/2009 Girish et al. 7,706.510 B2 4/2010 Ng 7,561,069 B2 7/2009 Horstemeyer 7,707,026 B2 4/2010 Liu 7,562,007 B2 7/2009 Hwang 7,707,027 B2 4/2010 Balchandran et al. 7,562,032 B2 7/2009 Abbosh et al. 7,707,032 B2 4/2010 Wang et al. 7,565,104 B1 7/2009 Brown et al. 7,707,221 B1 4/2010 Dunning et al. 7,565,380 B1 7/2009 Venkatachary 7,707,267 B2 4/2010 Lisitsa et al. 7,571,106 B2 8/2009 Cao et al. 7,710,262 B2 5 2010 Ruha 7,577,522 B2 8/2009 Rosenberg 7,711,129 B2 5, 2010 Lindahl et al. 7,580,551 B1 8/2009 Sriharietal. 7.711.550 B1 5/2010 Feinberg et al. 7,580,576 B2 8/2009 Wang et al. 7,711,565 B1 5, 2010 Gazdzinski 7,580,839 B2 8/2009 Tamura et al. 7,711,672 B2 5/2010 Au 7.584,093 B2 9, 2009 Potter et al. 7,712,053 B2 5, 2010 Bradford et al. 7.584.278 B2 9/2009 Rajarajan et al. 7,716,056 B2 5 2010 Weng 7.584,429 B2 9/2009 Fabritius 7,716,216 B1 5/2010 Harik et al. 7.593,868 B2 9/2009 Margiloffet al. 7,720,674 B2 5, 2010 Kaiser et al. 7,596,269 B2 9/2009 King et al. 7,720,683 B1 5, 2010 Vermeulen et al. 7,596,499 B2 9/2009 Anguera et al. 7,721,226 B2 5, 2010 Barabe et al. 7.596,606 B2 9/2009 Codignotto 7,721.301 B2 5 2010 Wong et al. 7,596,765 B2 9, 2009 Almás 7,724,242 B2 5, 2010 Hillis et al. 7,599,918 B2 10/2009 Shen et al. 7,725,307 B2 5/2010 Bennett 7,603,381 B2 10/2009 Burke et al. 7,725,318 B2 5, 2010 Gavalda et al. 7,609,179 B2 10/2009 Diaz-Gutierrez et al. 7,725,320 B2 5/2010 Bennett 7,610,258 B2 10/2009 Yuknewicz et al. 7,725,321 B2 5 2010 Bennett 7,613,264 B2 11/2009 Wells et al. 7,725,838 B2 5/2010 Williams 7,614,008 B2 11/2009 Ording 7,729,904 B2 6/2010 Bennett 7,617,094 B2 11/2009 Aoki et al. 7,729,916 B2 6, 2010 Coffman et al. 7,620.407 B1 1 1/2009 Donald et al. 7,734,461 B2 6/2010 Kwak et al. 7,620,549 B2 11/2009 Di Cristo et al. 7,735,012 B2 6/2010 Naik 7.623,119 B2 11/2009 Autio et al. 7,739,588 B2 6/2010 Reynar et al. 7,624,007 B2 11/2009 Bennett 7,742,953 B2 6/2010 King et al. 7.627,481 B1 12/2009 Kuo et al. 7,743,188 B2 6, 2010 Haitani et al. 7,630,901 B2 12/2009 Omi 7,747,616 B2 6/2010 Yamada et al. 7,633,076 B2 12/2009 Huppi et al. 7,752,152 B2 7/2010 Paek et al. 7,634.409 B2 12/2009 Kennewicket al. 7,756,868 B2 7/2010 Lee 7,634,413 B1 12/2009 Kuo et al. 7.757,173 B2 7/2010 Beaman 7,634,718 B2 12/2009 Nakajima 7,757,182 B2 T/2010 Elliott et al. 7,634,732 B1 12/2009 Blagsvedt et al. 7,761,296 B1 7/2010 Bakis et al. 7,636,657 B2 12/2009 Ju et al. 7,763.842 B2 7/2010 Hsu et al. 7,640,158 B2 12/2009 Detlefet al. 7,774,204 B2 8/2010 Mozer et al. 7,640,160 B2 12/2009 Cristo 7,774.388 B1 8/2010 Runchey 7,643,990 B1 1/2010 Bellegarda 7,777,717 B2 8, 2010 Fux et al. 7,647.225 B2 1/2010 Bennett et al. 7,778,432 B2 8/2010 Larsen 7,649,454 B2 1/2010 Singh et al. 7,778,595 B2 8, 2010 White et al. 7,649,877 B2 1/2010 Vieri et al. 7,778,632 B2 8, 2010 Kurlander et al. 7,653,883 B2 1/2010 Hotelling et al. 7,779,353 B2 8/2010 Grigoriu et al. 7,656,393 B2 2/2010 King et al. 7,779,356 B2 8, 2010 Griesmer 7,657,424 B2 2/2010 Bennett 7,779,357 B2 8, 2010 Naik 7,657,844 B2 2/2010 Gibson et al. 7,783,283 B2 8/2010 Kuusinen et al. 7,657,849 B2 2/2010 Chaudhri et al. 7,783,486 B2 8, 2010 Rosser et al. 7,663,607 B2 2/2010 Hotelling et al. 7,788,590 B2 8, 2010 Taboada et al. 7,664,558 B2 2/2010 Lindahl et al. 7,797.265 B2 9, 2010 Brinker et al. 7,664.638 B2 2/2010 Cooper et al. 7,797.269 B2 9, 2010 Rieman et al. 7,669,134 B1 2/2010 Christie et al. 7,797,331 B2 9/2010 Theimer et al. 7,672,841 B2 3/2010 Bennett 7,797,629 B2 9/2010 Fux et al. 7,672,952 B2 3/2010 Isaacson et al. 7,801,721 B2 9/2010 Rosart et al. 7,673,238 B2 3/2010 Girish et al. 7,801,728 B2 9/2010 Ben-David et al. 7,673,340 B1 3/2010 Cohen et al. 7,801,729 B2 9/2010 Mozer 7,676,026 B1 3/2010 Baxter, Jr. 7,805,299 B2 9/2010 Coifman 7,676.365 B2 3/2010 Hwang et al. 7,809,565 B2 10/2010 Coifman 7,676.463 B2 3/2010 Thompson et al. 7,809,569 B2 10/2010 Attwater et al. 7,679,534 B2 3/2010 Kay et al. 7,809,570 B2 10/2010 Kennewicket al. 7,680,649 B2 3/2010 Park 7,809,610 B2 10/2010 Cao US 9,318.108 B2 Page 13

(56) References Cited 7,983,917 T/2011 Kennewicket al. 7,983,919 T/2011 Conkie U.S. PATENT DOCUMENTS 7,983,997 T/2011 Allen et al. TO6, 12 7,984,062 T/2011 Dunning et al. 7,809,744 B2 10, 2010 Nevidomski et al. 7,986,431 T/2011 Emori et al. 7,818,165 B2 10, 2010 Carlgren et al. 7.987,151 T/2011 Schott et al. 7,818, 176 B2 10, 2010 Freeman et al. 7.987,244 T/2011 Lewis et al. 7,818,215 B2 10, 2010 King et al. 7.991,614 8, 2011 Washio et al. 7,818,291 B2 10, 2010 Ferguson et al. 7.992,085 8, 2011 Wang-Aryattanwanich et al. 7,818,672 B2 10, 2010 Mccormack et al. 7.996,228 8, 2011 Miller et al. 7,822,608 B2 10, 2010 Cross, Jr. et al. 7.996,589 8, 2011 Schultz et al. 7,823,123 B2 10, 2010 Sabbouh 7.996,769 8, 2011 Fux et al. 7,826,945 B2 11, 2010 Zhang et al. 7.996,792 8, 2011 AnZures et al. 7,827,047 B2 11, 2010 Anderson et al. 7.999,669 8, 2011 Singh et al. 7,831423 B2 11, 2010 Schubert 8,000,453 8, 2011 Cooper et al. 7,831.426 B2 11, 2010 Bennett 8,005,664 8, 2011 Hanumanthappa 7,831.432 B2 11, 2010 Bodin et al. 8,005,679 8, 2011 Jordan et al. 7,836,437 B2 11, 2010 Kacmarcik et al. 8,006,180 8, 2011 Tunning et al. 7,840.400 B2 11, 2010 Lavi et al. 8,015,006 9, 2011 Kennewicket al. 7,840.447 B2 11, 2010 Kleinrock et al. 8,015,011 9, 2011 Nagano et al. 7,840,581 B2 11, 2010 Ross et al. 8,015,144 9, 2011 Zheng et al. 7,840,912 B2 11, 2010 Elias et al. 8,018,431 9, 2011 Zehr et al. 7,848,924 B2 12, 2010 Nurminen et al. 8,019,271 9, 2011 Izdepski 7,848,926 B2 12, 2010 Goto et al. 8,024, 195 9, 2011 Mozer et al. 7,853.444 B2 12, 2010 Wang et al. 8,027,836 9, 2011 Baker et al. 7,853.445 B2 12, 2010 Bachenko et al. 8,031,943 10, 2011 Chen et al. 7,853,574 B2 12, 2010 Kraenzel et al. 8,032,383 10, 2011 Bhardwaj et al. 7,853,577 B2 12, 2010 Sundaresan et al. 8,036,901 10, 2011 Mozer 7,853,664 B1 12, 2010 Wang et al. 8,037,034 10, 2011 Plachta et al. 7,853,900 B2 12, 2010 Nguyen et al. 8,041,557 10, 2011 Liu 7,865,817 B2 1, 2011 Ryan et al. 8,041,570 10, 2011 Mirkovic et al. 7,869,999 B2 1, 2011 Amato et al. 8,041,611 10, 2011 Kleinrock et al. 7,870,118 B2 1, 2011 Jiang et al. 8,042,053 10, 2011 Darwish et al. 7,873,519 B2 1, 2011 Bennett 8,046,363 10, 2011 Cha et al. 7,873,654 B2 1, 2011 Bernard 8,050,500 11, 2011 Batty et al. 7,877,705 B2 1, 2011 Chambers et al. 8,055,502 11, 2011 Clark et al. 7,880,730 B2 2, 2011 Robinson et al. 8,055,708 11/2011 ChitsaZetal. 7,881.283 B2 2, 2011 Cormier et al. 8,060,824 11, 2011 Brownrigg et al. 7,881,936 B2 2, 2011 Longé et al. 8,064,753 11, 2011 Freeman 7,885,844 B1 2, 2011 Cohen et al. 8,065,143 11, 2011 Yanagihara 7,886,233 B2 2, 2011 Rainisto et al. 8,065,155 11, 2011 Gazdzinski 7,889, 184 B2 2, 2011 Blumenberg et al. 8,065,156 11, 2011 Gazdzinski 7,889,185 B2 2, 2011 Blumenberg et al. 8,068,604 11, 2011 Leeds et al. 7,890,330 B2 2, 2011 Ozkaragoz et al. 8,069,046 11, 2011 Kennewicket al. 7,890,652 B2 2, 2011 Bull et al. 8,069,422 11, 2011 Sheshagiri et al. 7,895,531 B2 2, 2011 Radtke et al. 8,073,681 12/2011 Baldwin et al. 7,899,666 B2 3, 2011 Varone 8,077,153 12/2011 Benko et al. 7,908,287 B1 3, 2011 Katragadda 8,078.473 12/2011 Gazdzinski 7,912,289 B2 3, 2011 Kansal et al. 8,082,153 12/2011 Coffman et al. 7,912,699 B1 3, 2011 Saraclaret al. 8,082,498 12/2011 Salamon et al. 7,912,702 B2 3, 2011 Bennett 8,090,571 1, 2012 Elshishiny et al. 7,912,720 B1 3, 2011 Hakkani-Tur et al. 8,095,364 1, 2012 Longé et al. 7.912,828 B2 3, 2011 Bonnet et al. 8,099,289 1, 2012 Mozer et al. 7,913,185 B1 3, 2011 Benson et al. 8,099.395 1, 2012 Pabla et al. 7,916,979 B2 3, 2011 Simmons 8,099.418 1, 2012 Inoue et al. 7,917,367 B2 3, 2011 Di Cristo et al. 8, 103,510 1, 2012 Sato 7.917497 B2 3, 2011 Harrison et al. 8, 107.401 1, 2012 John et al. 7,920,678 B2 4, 2011 Cooper et al. 8, 112,275 2, 2012 Kennewicket al. 7920,682 B2 4, 2011 Byrne et al. 8, 112,280 2, 2012 Lu 7920,857 B2 4, 2011 Lau et al. 8, 117,037 2, 2012 Gazdzinski 7,925,525 B2 4, 2011 Chin 8, 117,542 2, 2012 Radtke et al. 7,925,610 B2 4, 2011 Elbaz et al. 8,121413 2, 2012 Hwang et al. 7,929,805 B2 4, 2011 Wang et al. 8,121,837 2, 2012 Agapi et al. 7,930,168 B2 4, 2011 Weng et al. 8,122,094 2, 2012 Kotab 7,930, 183 B2 4, 2011 Odell et al. 8,122.353 2, 2012 Bouta 7,930, 197 B2 4, 2011 Ozzie et al. 8, 131,557 3, 2012 Davis et al. 7,936,339 B2 5, 2011 Marggraffet al. 8, 135,115 3, 2012 Hogg, Jr. et al. 7,941,009 B2 5, 2011 Li et al. 8,138,912 3, 2012 Singh et al. 7,945,470 B1 5, 2011 Cohen et al. 8, 140,335 3, 2012 Kennewicket al. 7,949,529 B2 5, 2011 Weider et al. 8, 140,567 3, 2012 Padovitz et al. 7,949,534 B2 5, 2011 Davis et al. 8, 150,694 4, 2012 Kennewicket al. 7,953,679 B2 5, 2011 Chidlovskii et al. 8, 150,700 4, 2012 Shin et al. 7,957,975 B2 6, 2011 Burns et al. 8, 155,956 4, 2012 Cho et al. 7,962, 179 B2 6, 2011 Huang 8,156,005 4, 2012 Vieri 7.974,844 B2 T/2011 Sumita 8, 160,883 4, 2012 Lecoeuche ...... TO4/270.1 7,974,972 B2 T/2011 Cao 8, 165,321 4, 2012 Paquier et al. 7.975,216 B2 T/2011 Woolfetal. 8, 165,886 4, 2012 Gagnon ...... G1 OL15, 26 7,983.478 B2 T/2011 Liu et al. 704/275 7,983,915 B2 T/2011 Knight et al. 8, 166,019 4, 2012 Lee et al. US 9,318.108 B2 Page 14

(56) References Cited 2001/0005859 A1 6/2001 Okuyama et al. 2001/0020259 A1 9/2001 Sekiguchi et al. U.S. PATENT DOCUMENTS 2001/0027396 A1 10, 2001 Sato 2001/00294.55 A1 10, 2001 Chin et al. 8, 170,790 B2 5, 2012 Lee et al. 2001/0030660 A1 10, 2001 Zainoulline 8.75370 B1 5/2012 Yamasani et al. 2001/0032080 A1 10, 2001 Fukada 8, 188,856 B2 5/2012 Singh et al. 2001/0041021 A1 1 1/2001 Boyle et al. 8, 190,359 B2 5, 2012 Bourne 2001/0042107 A1 11, 2001 Palm 8, 195,467 B2 6, 2012 MoZer et al. 2001/0044724 A1 11/2001 Hon et al. 8,200.495 B2 6, 2012 Braho et al. 2001/0047264 A1 11/2001 Roundtree 8,2009 B2 6/2012 Vanos et al. 2001.0056342 A1 12/2001 Piehn et al. 8,204.238 B2 6, 2012 Mozer 2001/0056347 A1 12/2001 Chazan et al. 8,205,788 B1 6/2012 Gazdzinski et al. 2002/0001395 A1 1/2002 Davis et al. 8,209,183 B1 6, 2012 Patel et al. 2002.0002039 A1 1/2002 Qureshey et al. 82195 B 7/2012 Neissen 2002.0002413 A1 1/2002 Tokue 8216.406 B2 7, 2012 Yu et al. 2002fOOO2461 A1 1/2002 Tetsumoto 8,219,407 B1 7/2012 Roy et al. 2002, 0004703 A1 1/2002 Gaspard, II 8,219,608 B2 7, 2012 Alsafadi et al. 2002fOO 10581 A1 1/2002 Euler et al. 8,224,649 B2 7, 2012 Chaudhari et al. 2002fOO 10584 A1 1/2002 Schultz et al. 8,239.207 B2 8/2012 Seligman et al. 2002/0010726 A1 1/2002 Rogson 8.255.217 B2 8, 2012 Stent et al. 2002/0010798 A1 1/2002 Ben-Shaul et al. 8,275,621 B2 9, 2012 Alewine et al. 2002fOO 13707 A1 1/2002 Shaw et al. 8,285,546 B2 10/2012 Reich 2002fOO 13784 A1 1/2002 Swanson 8.285.551 B2 10, 2012 GaZdzinski 2002fOO13852 A1 1/2002 Janik 8,285,553 B2 0/2012 Gazdzinski 2002fOO15024 A1 2/2002 Westerman et al. 8,290,777 B1 10/2012 Nguyen et al. 2002fOO15064 A1 2/2002 Robotham et al. 8.290.778 B2 10, 2012 GaZdzinski 2002, 0021278 A1 2/2002 Hinckley et al. 8.296.78 B2 0/2012 Gazdzinski 2002fOO26315 A1 2/2002 Miranda 8,296,146 B2 10/2012 GaZdzinski 2002fOO2645.6 A1 2/2002 Bradford 8.296.153 B2 10, 2012 GaZdzinski 2002fOO31254 A1 3/2002 Lantrip et al. 8.296.380 Bf 10/2012 Kellyet al. 2002/0031262 A1 3/2002 Imagawa et al. 8,296,383 B2 10/2012 Lindahl 2002fOO32048 A1 3/2002 Kitao et al. 8,300,801 B2 10/2012 Sweeney et al. 2002fOO32564 A1 3/2002 Ehsani et al. 8.301,456 B2 10, 2012 GaZdzinski 2002fOO32591 A1 3/2002 Mahaffy et al. 831834 B1 1/2012 Gazdzinski 2002/0032751 A1 3/2002 Bharadwaj 8,321,786 B2 11/2012 Lunati et al. 2002.0035467 A1 3/2002 Morimoto et al. 8.332.224 B2 12/2012 Di Cristo et al. 2002/0035469 A1 3/2002 Holzapfel 8.332.748 B1 12/2012 Karam 2002/0035474 A1 3/2002 Alpdemir 8,345,665 B2 1/2013 Vieri et al. 2002/0040359 A1 4/2002 Green et al. 8,352.183 B2 1/2013 Thota et al. 2002fOO42707 A1 4/2002 Zhao et al. 8,352,268 B2 1/2013 Naiketal. 2002/0045438 A1 4/2002 Tagawa et al. 8,352,272 B2 1/2013 Rogers et al. 2002fOO45961 A1 4/2002 Gibbs et al. 8,355,919 B2 1/2013 Silverman et al. 2002/0046025 A1 4/2002 Hain 8,359,234 B2 1/2013 Vieri 2002, 0046315 A1 4/2002 Miller et al. 8370,158 B2 2/2013 Gazdzinski 2002fOO52730 A1 5, 2002 Nakao 8,371,503 B2 2/2013 Gazdzinski 2002fOO5274.0 A1 5, 2002 Charlesworth et al. 8.374,871 B2 2/2013 Ehsani et al. 2002fOO52747 A1 5, 2002 Sarukkai 8,375.320 B2 2/2013 Kotleret al. 2002fOO54094 A1 5, 2002 Matsuda 8,380,504 B1 2/2013 Peden et al. 2002fOO55844 A1 5/2002 L’Esperance et al. 8,381,107 B2 2/2013 Rottler et al. 2002fOO55934 A1 5/2002 Lipscomb et al. 8,381,135 B2 2/2013 Hotelling et al. 2002/0059066 A1 5/2002 O'hagan 8,396,714 B2 3/2013 Rogers et al. 2002fOO59068 A1 5, 2002 Rose et al. 8.423,288 B2 4/2013 Stahl et al. 2002fOO65659 A1 5, 2002 Isono et al. 8428.758 B2 4/2013 Naik et al. 2002fOO65797 A1 5, 2002 Meidan et al. 8.447,612 B2 5 2013 Gazdzinski 2002fOO673O8 A1 6, 2002 Robertson 8,479,122 B2 7/2013 Hotelling et al. 2002fOO69063 A1 6/2002 Buchner et al. 8.489,599 B2 7, 2013 Bellotti 2002fOO69220 A1 6/2002 Tran 8.498,857 B2 7/2013 Kopparapu et al. 2002/007281.6 A1 6/2002 Shdema et al. 8,521,513 B2 8, 2013 Millett et al. 2002fOO72908 A1 6/2002 Case et al. 8,583,416 B2 11/2013 Huang et al. 2002fOO72914 A1 6/2002 Alshawi et al. 8,589,869 B2 11/2013 Wolfram 2002fOO77082 A1 6/2002 Cruickshank 8,595,004 B2 11/2013 Koshinaka 2002fOO77817 A1 6/2002 Atal 8,620,659 B2 12/2013 Di Cristo et al. 2002fOO78041 A1 6, 2002 Wu 8,645,137 B2 2/2014 Bellegarda et al. 2002 0080163 Al 62002 Morey 8,655,901 B1 2/2014 Li et al. 2002/0085037 A1 7/2002 Leavitt et al. 8,660,849 B2 2/2014 Gruber et al. 2002fOO87508 A1 7/2002 Hull et al. 8,660,970 B1 2/2014 Fiedorowicz 2002fO091511 A1 7/2002 Hellwig et al. 8,688,446 B2 4/2014 Yanagihara et al. 2002fO095286 A1 7, 2002 ROSS et al. 8,706,472 B2 4/2014 Ramerth et al. 2002fOO95290 A1 7/2002 Kahn et al. 8,719,006 B2 5/2014 Bellegarda et al. 2002fOO99547 A1 7/2002 Chu et al. 8,731,610 B2 5/2014 Appaji 2002.0099.552 A1 7/2002 Rubinet al. 8,760,537 B2 6/2014 Johnson et al. 2002/0101447 A1 8, 2002 Carro 8,768,693 B2 7/2014 Bellegarda et al. 2002/0103641 A1 8, 2002 Kuo et al. 8,775,931 B2 7, 2014 Fux et al. 2002/0103644 A1 8/2002 Brocious et al. 8,838,457 B2 * 9/2014 Cerra et al...... 704/275 2002/0103646 A1 8/2002 Kochanski et al. 8,880,405 B2 * 1 1/2014 Cerra et al. . 704/275 2002/0107684 A1 8, 2002 Gao 8,886,540 B2 * 1 1/2014 Cerra et al. . 704/270 2002.0109709 A1 8/2002 Sagar 8,943,423 B2 * 1/2015 Merrill et al...... 715,762 2002/0110248 A1 8, 2002 Kovales et al. 8,972,878 B2 3/2015 Mohler et al. 2002/01 1181.0 A1 8, 2002 Khan et al. US 9,318.108 B2 Page 15

(56) References Cited 2003/0078766 A1 4/2003 Appelt et al. 2003/0078780 A1 4/2003 Kochanski et al. U.S. PATENT DOCUMENTS 2003/0078969 A1 4/2003 Sprague et al. 2003/0079024 A1 4/2003 Hough et al. 2002fO116082 A1 8, 2002 Gudorf 2003/0079038 A1 4/2003 Robbinet al. 2002fO116171 A1 8, 2002 Russell 2003/0080991 A1 5/2003 Crow et al. 2002fO1161.85 A1 8/2002 Cooper et al. 2003/0083113 A1 5/2003 Chua et al. 2002fO116189 A1 8, 2002 Yeh et al. 2003/0083,878 A1 5/2003 Lee et al. 2002fO116420 A1 8, 2002 Allam et al. 2003, OO83884 A1 5/2003 Odinak et al. 2002/O120697 A1 8/2002 Generous et al. 2003/0084.350 AI 52003 Eibach et al. 20O2/O120925 A1 8/2002 Logan 2003/0O85870 A1 5/2003 Hinckley 2002/O122053 A1 9, 2002 Dutta et al. 2003, OO86699 A1 5/2003 Benyamin et al. 2002/O123894 A1 9, 2002 Woodward 2003, OO88414 A1 5/2003 Huang et al. 2002/O126097 A1 9, 2002 Savolainen 2003/0088421 A1 5/2003 Maes et al. 2002/O128827 A1 9, 2002 Bu et al. 2003/0090467 A1 5/2003 Hohl et al. 2002/O128840 Al 9, 2002 Hinde et al. 2003/0090474 A1 5/2003 Schaefer 2002/01290.57 A1 9/2002 Spielberg 2003/OO95096 A1 5/2003 Robbin et al. 2002/0133347 A1 9/2002 Schoneburg et al. 2003/0097210 A1 5.2003 Horst et al. 2002/0133348 A1 9, 2002 Pearson et al. 2003/0097379 A1 5.2003 Ireton 2002/0135565 A1 9, 2002 Gordon et al. 2003/0097408 Al 52003 Kageyama et al. 2002/0135618 A1 9, 2002 Maes et al. 2003/0098892 A1 5/2003 Hipakka 2002/0138254 A1 9, 2002 Isaka et al. 2003/0099.335 A1 5/2003 Tanaka et al. 2002/0138265 A1 9, 2002 Stevens et al. 2003/0101045 A1 5/2003 Moffatt et al. 2002/0138270 A1 9/2002 Bellegarda et al. 2003/01 15060 A1 6/2003 Junqua et al. 2002/01386.16 A1 9, 2002 Basson et al. 2003.01.15064 A1 6/2003 Gusler et al. 2002fO140679 A1 10, 2002 Wen 2003.01.15186 A1 6/2003 Wilkinson et al. 2002/0143533 A1 10, 2002 Lucas et al. 2003/0115552 A1 6/2003 Jahnke et al. 2002.0143542 A1 10, 2002 Eide 2003/01 17365 A1 6/2003 Shteyn 2002/0143551 A1 10, 2002 Sharma et al. 2003/0120494 Al 6, 2003 Jost et al. 2002.0143826 A1 10/2002 Day et al. 2003/O122787 A1 7/2003 Zimmerman et al. 2002fO151297 A1 10, 2002 Remboski et al. 2003/O125927 A1 7, 2003 Seme 2002/0152045 A1 10/2002 Dowling et al. 2003/O125955 A1 7, 2003 Arnold et al. 2002fO152255 A1 10, 2002 Smith et al. 2003/O126559 A1 7, 2003 Fuhrmann 2002fO15416.0 A1 10, 2002 Hosokawa 2003.01288.19 A1 7, 2003 Lee et al. 2002/0161865 A1 10/2002 Nguyen 2003/033694 A1 7, 2003 Yeo 2002/0163544 A1 11/2002 Baker et al. 2003/0134678 A1 7, 2003 Tanaka 2002/0164000 A1 11/2002 Cohen et al. 2003. O135740 A1 7/2003 Tallmor et al. 2002fO165918 A1 11, 2002 Bettis 2003. O140088 A1 7/2003 Robinson et al. 2002fO167534 A1 11, 2002 Burke 2003. O144846 A1 7/2003 Denenberg et al. 2002/0169592 A1 1 1/2002 Aityan 2003. O145285 A1 7/2003 Miyahira et al. 2002/0169605 A1 11/2002 Damiba et al. 2003. O147512 A1 8, 2003 Abburi 2002/0173273 Al 1 1/2002 Spurgatet al. 2003/01495.57 A1 8/2003 Cox et al. 2002/0173889 A1 11/2002 Odinak et al. 2003/O1495.67 A1 8, 2003 Schmitz et al. 2002/0173961 A1 11/2002 Guerra 2003/O149978 A1 8, 2003 Plotnick 2002/0173962 A1 1 1/2002 Tang et al. 2003. O152203 A1 8/2003 Berger et al. 2002/0173966 A1 11, 2002 Henton 2003. O152894 A1 8, 2003 Townshend 2002/0177993 A1 11, 2002 Veditz et al. 2003. O154081 A1 8, 2003 Chu et al. 2002fO184027 A1 12/2002 Brittan et al. 2003/O157968 A1 8/2003 Boman et al. 2002/0184189 A1 12/2002 Hay et al. 2003/O158735 A1 8/2003 Yamada et al. 2002/018.9426 A1 12/2002 Hirade et al. 2003. O158737 A1 8, 2003. CsicSatka 2002/019 1029 A1 12/2002 Gillespie et al. 2003.0160702 A1 8, 2003 Tanaka 2002/0193996 Al 12/2002 Squibbs et al. 2003/0160830 A1 8/2003 Degross 2002fO198714 A1 12, 2002 Zhou 2003. O16331.6 A1 8/2003 Addison et al. 2002fO1987.15 A1 12/2002 Belrose 2003. O164848 A1 9, 2003 Dutta et al. 2003/0001881 A1 1/2003. Mannheimer et al. 2003/0167167 Al 9, 2003 Gong 2003,0002632 A1 1/2003 Bhogal et al. 2003.0167318 A1 9, 2003 Robbin et al. 2003/0013483 A1 1/2003 AuSems et al. 2003.0167335 A1 9, 2003 Alexander 2003, OO16770 A1 1/2003 Trans et al. 2003/0171928 A1 9, 2003 Falcon et al. 2003/002O760 A1 1/2003 Takatsu et al. 2003/0171936 A1 9, 2003 Sallet al. 2003/00264.02 A1 2/2003 Clapper 2003/017483.0 A1 9/2003 Boyer et al. 2003/0028380 A1 2/2003 Freeland et al. 2003/0177.046 A1 9/2003 Socha-Leialoha et al. 2003/0033153 A1 2/2003 Olson et al. 2003/0179222 A1 9, 2003 Noma et al. 2003/0033214 A1 2/2003 Mikkelsen et al. 2003/01821 15 A1 9/2003 Malayath et al. 2003/0037073 A1 2/2003 Tokuda et al. 2003/0182131 A1 9, 2003 Arnold et al. 2003, OO37254 A1 2/2003 Fischer et al. 2003/0187655 Al 10/2003 Dunsmuir 2003/004O908 A1 2/2003 Yang et al. 2003. O187844 A1 10, 2003 Li et al. 2003/0046075 A1 3/2003 Stone 2003/O187925 A1 10/2003 Inala et al. 2003/0046401 A1 3/2003 Abbott et al. 2003/0188005 A1 10, 2003 Yoneda et al. 2003/0046434 A1 3/2003 Flanagin et al. 2003/0188192 A1 10/2003 Tang et al. 2003/0050781 A1 3/2003 Tamura et al. 2003/0190.074 A1 10, 2003 Loudon et al. 2003/005113.6 A1 3f2003 Curtis et al. 2003. O191645 A1 10, 2003 Zhou 2003/0055537 A1 3, 2003. Odinak et al. 2003/0193481 A1 10/2003 Sokolsky 2003/0061317 A1 3f2003 Brown et al. 2003/0194080 A1 10, 2003 Michaelis et al. 2003, OO61570 A1 3, 2003 Hatori et al. 2003. O195741 A1 10, 2003 Mani et al. 2003/0063073 A1 4/2003 Geaghan et al. 2003/0197736 A1 10/2003 Murphy 2003/0074195 A1 4/2003 Bartosik et al. 2003. O197744 A1 10, 2003 Irvine 2003/0074198 A1 4/2003 Sussman 2003/0200858 A1 10, 2003 Xie 2003/0074457 A1 4, 2003 Kluth 2003/0204392 A1 10/2003 Finnigan et al. 2003/0076301 A1 4, 2003 TSuk et al. 2003/0204492 A1 10, 2003 Wolfetal. US 9,318.108 B2 Page 16

(56) References Cited 2004/0135774 A1 7, 2004 La Monica 2004/O13651.0 A1 7/2004 Vander Veen U.S. PATENT DOCUMENTS 2004/O138869 A1 7/2004 Heinecke 2004/0145607 A1 7/2004 Alderson 2003/0208756 A1 11/2003 Macrae et al. 2004/0153306 Al 8, 2004 Tanner et al. 2003/0210266 A1 1 1/2003 Cragun et al. 2004/0160419 A1 8/2004 Padgitt 2003/0212961 A1 11/2003 Soin et al. 2004O162741 A1 8, 2004 Flaxer et al. 2003/0214519 A1 11, 2003 Smith et al. 2004/0174399 A1 9, 2004 Wu et al. 2003/0224760 A1 12/2003 Day 2004/0174434 A1 9, 2004 Walker et al. 2003/0228863 A1 12/2003 Vander Veen et al. 2004/0176958 A1 9/2004 Salmenkaita et al. 2003/0228909 A1 12/2003 Tanaka et al. 2004/017.7319 Al 9, 2004 Horn 2003,0229490 A1 12, 2003 Etter 2004/0178994 A1 9/2004 Kairls, Jr. 2003/0229616 A1 12/2003 Wong 2004/O183833 A1 9, 2004 Chua 2003/0233230 A1 12/2003 Ammicht et al. 2004/0186713 A1 9, 2004 Gomas et al. 2003/0233237 A1 12/2003 Garside et al. 2004/01867.14 Al 9, 2004 Baker 2003/0233240 A1 12/2003 Kaatrasalo 2004/0186777 A1 9/2004 Margiloff et al. 2003/0234824 A1 12, 2003 Litwiler 2004/0193398 Al 9, 2004 Chu et al. 2003/0236663 A1 12/2003 Dimitrova et al. 2004/0193420 A1 9, 2004 Kennewicket al. 2004/0001396 A1 1/2004 Keller et al. 2004.0193421 A1 9, 2004 Blass 2004/0006467 A1 1/2004 Anisimovich et al. 2004/0193426 A1 9, 2004 Maddux et al. 2004, OO12556 A1 1/2004 Yong et al. 2004/0196256 A1 10, 2004 Wobbrocket al. 2004, OO13252 A1 1/2004 Craner 2004/O198436 A1 10, 2004 Alden 2004/0021676 A1 2/2004 Chen et al. 2004/0199.375 A1 10, 2004 Ehsani et al. 2004/0022369 A1 2/2004 Vitikainen ...... HO4M 3/533 20040199387 Al 10/2004 Wang et al. 379 (67.1 2004/01996.63 A1 10, 2004 Horvitz et al. 2004/0022373 A1 2/2004 Suder et al. 2004/0203520 Al 10/2004 Schirtzinger et al. 2004/0023643 A1 2/2004 Vander Veen et al. 2004/0205151 Al 10/2004 Sprigget al. 2004/0030556 A1 2/2004 Bennett 2004/0205671 A1 10, 2004 Sukehiro et al. 2004/OO30559 A1 2/2004 Payne et al. 2004/0208302 A1 10, 2004 Urban et al. 2004/0030996 A1 2/2004 Van Liempdet al. 2004/0210634 A1 10, 2004 Ferrer et al. 2004.0036715 A1 2/2004 Warren 2004/0213419 A1 10, 2004 Varma et al. 2004/0048627 A1 3f2004 Olvera-Hernandez 2004/0215731 A1 10/2004 Tzann-en Szeto 2004.0049391 A1 3/2004 Polanyi et al. 2004/0216049 A1 10, 2004 Lewis et al. 2004/0051729 A1 3/2004 Borden, IV 2004/0218451 Al 11/2004 Said et al. 2004/0052338 A1 3/2004 Celi, Jr. et al. 2004/O220798 A1 11, 2004 Chi et al. 2004.0054530 A1 3, 2004 Davis et al. 2004/0223485 A1 11/2004 Arellano et al. 2004.0054533 A1 3/2004 Bellegarda 2004/0223599 A1 11/2004 Bear et al. 2004.0054534 A1 3/2004 Junqua 2004/0224.638 A1 11/2004 Fadell et al. 2004.0054535 A1 3, 2004 Mackie et al. 2004/0225501 A1 11/2004 Cutaia et al. 2004.0054541 A1 3/2004 Kryze et al. 2004/0225650 Al 1 1/2004 Cooper et al. 2004/0054690 A1 3f2004 Hillerbrand et al. 2004/0225746 A1 11, 2004 Niell et al. 2004/0055446 A1 3, 2004 Robbin et al. 2004/0230637 A1* 11/2004 Lecoueche et al...... TO9/200 2004/0056899 A1 3/2004 Sinclair, II et al. 2004/0236778 A1 1 1/2004 Junqua et al. 2004/0059577 A1 3/2004 Pickering 2004/0242286 All 12/2004 Benco et al. 2004/0059790 A1 3f2004 Austin-Lane et al. 2004/0243412 A1 12/2004 Gupta et al. 2004/006 1717 A1 4/2004 Menon et al. 2004/0243419 A1 12/2004 Wang 2004/0062367 A1 4/2004 Fellenstein et al. 2004/02496.29 A1 12, 2004 Webster 2004/0064593 A1 4/2004 Sinclair et al. 2004/0249667 A1 12, 2004 OOn 2004, OO69122 A1 4/2004 Wilson 2004/02521 19 A1 12/2004 Hunleth et al. 2004/0070567 A1 4/2004 Longe et al. 2004/0252604 A1 12/2004 Johnson et al. 2004/0070612 A1 4/2004 Sinclair et al. 2004/0252966 Al 12/2004 Holloway et al. 2004/OO73427 A1 4/2004 Moore 2004/0254791 A1 12/2004 Coifman et al. 2004f0073428 A1 4/2004 Zlokarnik et al. 2004/0254792 Al 12/2004 Busayapongchai et al. 2004/0076086 A1 4/2004 Keller et al. 2004/0257432 A1 12/2004 Girish et al. 2004/0078382 A1 4/2004 Mercer et al. 2004/0259536 A1 12/2004 Keskar et al. 2004/0085162 A1 5/2004 Agarwal et al. 2004/0260438 Al 12/2004 Chernetsky et al. 2004/0085368 A1 5/2004 Johnson, Jr. et al. 2004/0261023 A1 12, 2004 Bier 2004/0086120 A1 5/2004 Akins, III et al. 2004/O262051 A1 12, 2004 Carro 2004/00932.13 A1 5, 2004 Conkie 2004/026363.6 A1 12/2004 Cutler et al. 2004/00932.15 A1 5/2004 Gupta et al. 2004/0267825 A1 12, 2004 Novak et al. 2004/0094.018 A1 5, 2004 Ueshima et al. 2004/0268253 A1 12/2004 Demello et al. 2004/0096105 A1 5/2004 Holtsberg 2004/0268262 Al 12/2004 Gupta et al. 2004/0098.250 A1 5, 2004 Kimchi et al. 2005/OOO2507 A1 1/2005 Timmins et al. 2004/0100479 A1 5/2004 Nakano et al. 2005, OO 10409 A1 1/2005 Hull et al. 2004/0106432 A1 6/2004 Kanamori et al. 2005, OO12723 A1 1/2005 Pallakoff 2004/O107169 A1 6, 2004 Lowe 2005, OO15254 A1 1/2005 Beaman 2004/011 1266 A1 6/2004 Coorman et al. 2005, OO15772 A1 1/2005 Saare et al. 2004/011 1332 A1 6, 2004 Baar et al. 2005/00221 14 A1 1/2005 Shanahan et al. 20040114731 A1 6, 2004 Gillett et al. 2005.0024341 A1 2/2005 Gillespie et al. 2004.0122656 A1 6, 2004 Abir 2005.0024345 A1 2/2005 Eastty et al. 2004/O122664 A1 6/2004 Lorenzo et al. 2005, OO27385 A1 2/2005 Yueh 2004/O124583 A1 7, 2004 Lands 2005/OO3O175 A1 2/2005 Wolfe 2004/O125088 A1 7/2004 Zimmerman et al. 2005/0031106 A1 2/2005 Henderson 2004/0125922 A1 7/2004 Specht 2005/0033582 A1 2/2005 Gadd et al. 2004/O127198 A1 7/2004 Roskind et al. 2005/0033771 A1 2/2005 Schmitter et al. 2004/O127241 A1 7, 2004 Shostak 2005.0034164 A1 2/2005 Sano et al. 2004/O128137 A1 7, 2004 Bush et al. 2005/0038657 A1 2/2005 Roth et al. 2004/O133817 A1 7, 2004 Choi 2005, OO39141 A1 2/2005 Burke et al. 2004/0135701 A1 7/2004 Yasuda et al. 2005/0043946 A1 2/2005 Ueyama et al. US 9,318.108 B2 Page 17

(56) References Cited 2005/02O1572 A1 9, 2005 Lindahl et al. 2005/0202854 A1 9, 2005 Kortum et al. U.S. PATENT DOCUMENTS 2005/0203747 A1* 9, 2005 Lecoeuche ...... TO4/270.1 2005/02O3991 A1 9/2005 Kawamura et al. 2005, 0043949 A1 2/2005 Roth et al. 2005/0209848 A1 9, 2005 Ishii 2005/0044569 A1 2/2005 Marcus 2005/0210394 A1 9, 2005 Crandall et al. 2005/0045373 A1 3, 2005 Born 2005/0216331 A1 9, 2005 Ahrens et al. 2005.0049880 A1 3, 2005 Roth et al. 2005/0222843 Al 10/2005 Kahn et al. 2005/0055212 A1 3/2005 Nagao 2005/0222973 A1 10, 2005 Kaiser 2005/0055403 A1 3f2005 Brittan ...... TO9,206 2005/0228665 A1 10/2005 Kobayashi et al. 2005/0058438 A1 3/2005 Hayashi 2005/0245243 Al 1 1/2005 Zuniga 2005/006O155 A1 3, 2005 Chu et al. 2005/0246350 A1 11/2005 Canaran 2005, 0071.165 A1 3, 2005 Hofstader et al. 2005/0246365 A1 11/2005 Lowles et al. 2005, 0071332 A1 3/2005 Ortega et al. 2005/0246726 A1 11/2005 Labrou et al. 2005, 0071.437 A1 3/2005 Bear et al. 2005/0267738 A1 12/2005 Wilkinson et al. 2005, OO74113 A1 4/2005 Mathew et al. 2005/0267757 A1 12/2005 Iso-Sipila et al. 2005/0080613 A1 4/2005 Colledge et al. 2005/0271216 A1 12/2005 Lashkari 2005, 0080620 A1 4/2005 Rao et al. 2005/0273337 A1 12/2005 Erell et al. 2005, 0080625 A1 4/2005 Bennett et al. 2005/0273626 A1 12/2005 Pearson et al. 2005, 0080632 A1 4, 2005 Endo et al. 2005/0278297 Al 12/2005 Nelson 2005/0080780 A1 4/2005 Colledge et al. 2005/0278643 Al 12/2005 Ukai et al. 2005/0O86059 A1 4/2005 Bennett 2005/0278647 A1 12/2005 Leavitt et al. 2005, OO862.55 A1 4, 2005 Schran et al. 2005/0283364 Al 12/2005 Longe et al. 2005, OO86605 A1 4, 2005 Ferrer et al. 2005/0283726 Al 12/2005 Lunati 2005/0091118 A1 4/2005 Fano 2005/0283729 A1 12/2005 Morris et al. 2005/0094475 A1 5, 2005 Nao 2005/0288934 A1 12/2005 Omi 2005.0099398 A1 5, 2005 Garside et al. 2005/0288936 Al 12/2005 Busayapongchai et al. 2005, 01 00214 A1 5/2005 Zhang et al. 2005/02894.63 A1 12/2005 Wu et al. 2005, 0102144 A1 5/2005 Rapoport 2006, OOO1652 A1 1/2006 Chiu et al. 2005, 0102614 A1 5/2005 Brockett et al. 2006/0004570 A1 1/2006 Ju et al. 2005, 0102625 A1 5, 2005 Lee et al. 2006,0004744 A1 1/2006 Nevidomski et al. 2005, 0105712 A1 5, 2005 Williams et al. 2006,0007174 A1 1/2006 Shen 2005, 0108001 A1 5/2005 Aarskog 2006, OOO9973 A1 1, 2006 Nguyen et al. 2005, 0108017 A1 5, 2005 ESSer et al. 2006, OO13414 A1 1/2006 Shih 2005, 0108074 A1 5, 2005 Boechlet al. 2006/0015341 A1 1/2006 Baker 2005, 0108338 A1 5, 2005 Simske et al. 2006, OO15819 A1 1/2006 Hawkins et al. 2005, 0108344 A1 5/2005 Tafoya et al. 2006001844.6 A1 1/2006 Schmandt et al. 2005.0114124 A1 5, 2005 Liu et al. 2006, OO18492 A1 1/2006 Chiu et al. 2005/011414.0 A1 5/2005 Brackett et al. 2006.0020890 A1 1/2006 Kroll et al. 2005, 011989.0 A1 6, 2005 Hirose 2006, OO25999 A1 2/2006 Feng et al. 2005/01 19897 A1 6, 2005 Bennett et al. 2006, OO26233 A1 2/2006 Tenembaum et al. 2005/0125216 A1 6/2005 Chitrapura et al. 2006/0026521 A1 2/2006 Hotelling et al. 2005/O125235 A1 6/2005 Lazay et al. 2006, OO26535 A1 2/2006 Hotelling et al. 2005/O131951 A1 6/2005 Zhang et al. 2006, OO26536 A1 2/2006 Hotelling et al. 2005, 0132301 A1 6, 2005 Ikeda 2006, OO33724 A1 2/2006 Chaudhri et al. 2005. O136949 A1 6, 2005 Barnes, Jr. 2006.0035632 A1 2/2006 Sorwari et al. 2005/O138305 A1 6, 2005 Zeliner 2006.0036946 A1 2/2006 Radtke et al. 2005, 0140504 A1 6, 2005 Marshall et al. 2006,0041424 A1 2/2006 Todhunter et al. 2005/0143972 A1 6/2005 Gopalakrishnan et al. 2006,0041431 A1 2/2006 Maes 2005/0144003 A1 6/2005 Iso-Sipila 2006/0047632 A1 3/2006 Zhang 2005. O144070 A1 6, 2005 Cheshire 2006/0050865 A1 3/2006 Kortum et al. 2005/0144568 A1 6/2005 Gruen et al. 2006, OO52141 A1 3, 2006 Suzuki 2005. O148356 A1 7/2005 Ferguson et al. 2006.00533.65 A1 3/2006 Hollander et al. 2005. O149214 A1 7, 2005 Yoo et al. 2006.0053379 A1 3/2006 Henderson et al. 2005. O149330 A1 7, 2005 Katae 2006/0053387 A1 3/2006 Ording 2005/0149332 A1 7/2005 Kuzunuki et al. 2006/0058999 A1 3/2006 Barker et al. 2005. O14951.0 A1 7, 2005 Shafrir 2006/0059.437 A1 3, 2006 Conklin 2005. O152558 A1 7, 2005 Van Tassel 2006, OO60762 A1 3, 2006 Chan et al. 2005. O152602 A1 7, 2005 Chen et al. 2006/0061488 A1 3/2006 Dunton 2005/O154578 A1 7/2005 Tong et al. 2006, OO67535 A1 3/2006 Culbert et al. 2005/0154591 A1* 7/2005 Lecoeuche ...... TO4/270.1 2006/0067536 A1 3/2006 Culbert et al. 2005/O1599.39 A1 7/2005 Mohler et al. 2006, OO69567 A1 3/2006 Tischer et al. 2005/0162395 A1 7, 2005 Unruh 2006, OO69664 A1 3/2006 Ling et al. 2005. O165607 A1 7, 2005 DiFabbrizio et al. 2006/0072248 A1 4/2006 Watanabe et al. 2005/0166153 A1 7/2005 Eytchison et al. 2006, OO72716 A1 4, 2006 Pham 2005/0177445 A1 8, 2005 Church 2006, OO74628 A1 4/2006 Elbaz et al. 2005/O181770 A1 8, 2005 Helferich 2006, OO74660 A1 4/2006 Waters et al. 2005/0182616 A1 8/2005 Kotipalli 2006/0074674 A1 4/2006 Zhang et al. 2005, 0182627 A1 8/2005 Tanaka et al. 2006, OO747SO A1 4/2006 Clark et al. 2005, 0182628 A1 8, 2005 Choi 2006,0074898 A1 4/2006 Gavalda et al. 2005/0182629 A1 8/2005 Coorman et al. 2006, OO77055 A1 4, 2006 Basir 2005, 0182630 A1 8, 2005 Miro et al. 2006, 0080.098 A1 4/2006 Campbell 2005/O182765 A1 8/2005 Liddy 2006/0085.187 A1 4/2006 Barquilla 2005/O187773 A1 8, 2005 Filoche et al. 2006/00854.65 A1 4/2006 Nori et al. 2005/O19097O A1 9, 2005 Griffin 2006/0O85757 A1 4/2006 Andre et al. 2005, 01928O1 A1 9, 2005 Lewis et al. 2006/0095265 A1 5, 2006 Chu et al. 2005/O195077 A1 9, 2005 Mcculloch et al. 2006/0095790 A1 5/2006 Nguyen et al. 2005/O195429 A1 9, 2005 Archbold 2006/0095846 A1 5, 2006 Nurmi 2005/O196733 A1 9, 2005 Budra et al. 2006/0095848 A1 5, 2006 Naik US 9,318.108 B2 Page 18

(56) References Cited 2006/0252457 A1 1 1/2006 Schrager 2006/0253210 A1 1 1/2006 Rosenberg U.S. PATENT DOCUMENTS 2006/0253787 A1 1 1/2006 Fogg 2006, O256934 A1 11, 2006 Mazor 2006/0097991 A1 5/2006 Hotelling et al. 2006/0262876 Al 11/2006 LaDue 2006, O100848 A1 5, 2006 COZZi et al. 2006/0265208 A1 11/2006 ASSadollahi 2006, O100849 A1 5, 2006 Chan 2006/0265503 A1 11/2006 Jones et al. 2006/0101354 A1 5, 2006 Hashimoto et al. 2006/0265648 A1 11/2006 Rainisto et al. 2006, O103633 A1 5, 2006 Gioeli 2006/0271627 A1 1 1/2006 Szczepanek 2006/0106592 A1 5/2006 Brockett et al. 2006/0274051 Al 12/2006 Longeet al. 2006/0106594 A1 5, 2006 Brockett et al. 2006/0274905 A1 12/2006 Lindahl et al. 2006/0106595 A1 5, 2006 Brockett et al. 2006/0277.058 A1 12/2006 Jimaev et al. 2006/011 1906 A1 5/2006 Cross et al. 2006/0282264 Al 12/2006 Denny et al. 2006/011 1909 A1 5, 2006 Maes et al. 2006/0282415 A1 12/2006 Shibata et al. 2006/01 16874 A1 6/2006 Samuelsson et al. 2006/0286,527 Al 12/2006 Morel 2006/01 16877 A1 6/2006 Pickering et al. 2006/0288024 A1 12/2006 Braica 2006/01 17002 A1 6, 2006 Swen 2006/0291666 A1 12/2006 Ball et al. 2006/01 19582 A1 6/2006 Ng et al. 2006/0293876 A1 12/2006 Kamatani et al. 2006, O122834 A1 6, 2006 Bennett 2006/0293880 Al 12/2006 Elshishiny et al. 2006, O12283.6 A1 6, 2006 Cross et al. 2006, O293886 A1 12/2006 Odell et al. 2006/0129929 A1 6, 2006 Weber et al. 2007/0003026 A1 1/2007 Hodge et al. 2006/0132812 A1 6/2006 Barnes et al. 2007/0004451 A 1 1/2007 Anderson 2006/0136213 A1 6/2006 Hirose et al. 2007,0005849 A1 1/2007 Oliver 2006, O141990 A1 6, 2006 Zak et al. 2007, OOO6098 A1 1/2007 Krumm et al. 2006.0143007 A1 6, 2006 Koh et al. 2007.0011154 A1 1/2007 Musgrove et al. 2006/0143576 A1 6/2006 Gupta et al. 2007/00 16563 A1 1/2007 Omoigui 2006, O148520 A1 7/2006 Baker et al. 2007, OO16865 A1 1/2007 Johnson et al. 2006/0150.087 A1 7/2006 Cronenberger et al. 2007/0021956 A1 1/2007 Quet al. 2006, O152496 A1 7/2006 Knaven 2007/0022380 A1 1/2007 Swartz et al. 2006, O153040 A1 7/2006 Girish et al. 2007/0O25704 A1 2/2007 Tsukazaki et al. 2006/0156252 A1 7/2006 Sheshagiri et al. 2007/0026852 A1 2/2007 Logan et al. 2006/0156307 A1 7/2006 Kunjithapatham et al. 2007/0027732 A1 2/2007 Hudgens 2006/0161870 A1 7/2006 Hotelling et al. 2007/0028009 A1 2/2007 Robbin et al. 2006/0161871 A1 7/2006 Hotelling et al. 2007/0032247 Al 2, 2007 Shaffer et al. 2006/0161872 A1 7/2006 Rytivaara et al. 2007/0033003 Al 2, 2007 Morris 2006/0165105 A1 7/2006 Shenfield et al. 2007/0033026 A1 2/2007 Bartosik et al. 2006.0167676 A1 7/2006 Plumb 2007.00361.17 A1 2, 2007 Taube et al. 2006, O168150 A1 7/2006 Naik et al. 2007.0036286 A1 2/2007 Champlin et al. 2006, O168507 A1 7/2006 Hansen 2007/0038436 A1 2/2007 Cristo et al. 2006/0168539 A1 7/2006 Hawkins et al. 2007/0O38609 A1 2/2007 Wu 2006/0172720 A1 8, 2006 Islam et al. 2007/0040813 A1 2/2007 Kushler et al. 2006/0174207 A1 8/2006 Deshpande 2007/0041361 A1 2/2007 Iso-Sipila. 2006/0178868 A1 8/2006 Billerey-Mosier 2007/0043568 A1 2/2007 Dhanakshirur et al. 2006/0181519 A1 8/2006 Vernier et al. 2007/0044038 Al 2, 2007 Horentrup et al. 2006/0183466 A1 8/2006 Lee et al. 2007, 0046641 A1 3, 2007 Lim 2006, O184886 A1 8/2006 Chung et al. 2007/0047719 A1 3/2007 Dhawan et al. 2006/O187073 A1 8, 2006 Lin et al. 2007/005O184 A1 3/2007 Drucker et al. 2006, O190269 A1 8, 2006 Tessel et al. 2007/005O191 A1 3/2007 Weider et al. 2006/O190577 A1 8, 2006 Yamada 2007/0050393 A1 3/2007 Vogel et al. 2006, O193518 A1 8/2006 Dong 2007/0050712 A1 3, 2007 Hull et al. 2006, O195206 A1 8, 2006 Moon et al. 2007/0052586 A1 3/2007 Horstemeyer 2006/0195323 A1 8/2006 Monne et al. 2007/0055493 A1 3, 2007 Lee 2006/O197753 A1 9/2006 Hotelling 2007/0055508 A1 3, 2007 Zhao et al. 2006/O197755 A1 9/2006 Bawany 2007/0055514 A1 3/2007 Beattie et al. 2006/0200253 A1 9/2006 Hoffberg et al. 2007/005.5525 A1 3/2007 Kennewicket al. 2006/0200342 A1 9, 2006 Corston-Oliver et al. 2007/0055529 A1 3/2007 Kanevsky et al. 2006/020.0347 A1 9, 2006 Kim et al. 2007/0058832 A1 3/2007 Huget al. 2006/0205432 A1 9, 2006 Hawkins et al. 2007/0060107 A1 3/2007 Day 2006/0206454 A1 9, 2006 Forstall et al. 2007/0061487 A1 3/2007 Moore et al. 2006/0212415 A1 9, 2006 Backer et al. 2007/0061712 A1 3/2007 Bodin et al. 2006/0217967 A1 9, 2006 Goertzen et al. 2007/006 1754 A1 3/2007 Ardhanarietal. 2006/0221738 A1 10, 2006 Park et al. 2007/0067.173 A1 3/2007 Bellegarda 2006/0221788 A1 10, 2006 Lindahl et al. 2007/0067272 A1 3/2007 Flynt et al. 2006/0224.570 A1 10/2006 Quiroga et al. 2007/OO73540 A1 3, 2007 Hirakawa et al. 2006/0229802 A1 10/2006 Vertelney et al. 2007, OO73541 A1 3, 2007 Tian 2006/0229870 A1 10, 2006 Kobal 2007/OO73745 A1 3/2007 Scott et al. 2006/0229876 A1 10, 2006 Aaron et al. 2007/0075965 A1 4/2007 Huppiet al. 2006/0230410 A1 10/2006 Kurganov et al. 2007/0079027 A1 4/2007 Marriott et al. 2006/0234680 A1 10, 2006 Doulton 2007, 0080936 A1 4/2007 Tsuk et al. 2006/0235550 A1 10, 2006 CsicSatka et al. 2007, 0083467 A1 4, 2007 Lindahl et al. 2006/0235700 A1 10/2006 Wong et al. 2007, 0083623 A1 4/2007 Nishimura et al. 2006/0235841 A1 10, 2006 Betz et al. 2007/0088556 A1 4, 2007 Andrew 2006/0236262 A1 10, 2006 Bathiche et al. 2007/0089132 A1 4/2007 Qureshey et al. 2006/0239419 A1 10/2006 Joseph et al. 2007/0089135 A1 4/2007 Qureshey et al. 2006/02394.71 A1 10, 2006 Mao et al. 2007/0093.277 A1 4, 2007 Cavacuiti et al. 2006/0240866 A1 10, 2006 Eilts et al. 2007/0094026 A1 4/2007 Ativanichayaphong et al. 2006/0242190 A1 10, 2006 Winek 2007/0098195 A1 5, 2007 Holmes 2006/0246955 A1 11/2006 Nirhamo et al. 2007, 0100206 A1 5, 2007 Lin et al. 2006/0247931 A1 1 1/2006 Caskey et al. 2007, 0100602 A1 5, 2007 Kim US 9,318.108 B2 Page 19

(56) References Cited 2007/0229.323 A1 10, 2007 Plachta et al. 2007/0230729 A1 10/2007 Naylor et al. U.S. PATENT DOCUMENTS 2007/0233490 A1 10, 2007 Yao 2007/0233497 A1 10, 2007 Paek et al. 2007/010.0619 A1 5, 2007 Purho et al. 2007/0233692 A1 10/2007 Lisa et al. 2007/010.0635 A1 5/2007 Mahajan et al. 2007/0233725 A1 10, 2007 Michmerhuizen et al. 2007, 0100709 A1 5, 2007 Lee et al. 2007/0238488 A1 10, 2007 Scott 2007/0100790 A1 5/2007 Cheyer et al. 2007/0238489 A1 10/2007 Scott 2007, 0100883 A1 5, 2007 Rose et al. 2007/0238520 A1 10/2007 Kacmarcik 2007/0106512 A1 5, 2007 Acero et al. 2007/0239.429 A1 10, 2007 Johnson et al. 2007/0106513 A1 5, 2007 Boilot et al. 2007,024.0043 A1 10, 2007 Fux et al. 2007/0106674 A1 5/2007 Agrawal et al. 2007/0240044 A1 10, 2007 Fux et al. 2007/01 16195 A1 5/2007 Thompson et al. 2007/0240045 A1 10, 2007 Fux et al. 2007/01 18377 A1 5, 2007 Badino et al. 2007/0244702 AI 10/2007 Kahn et al. 2007/01 18378 A1 5/2007 Skuratovsky 2007/0247441 A1 10, 2007 Kim et al. 2007, 0121846 A1 5/2007 Altberg et al. 2007/0255435 Al 11/2007 Cohen et al. 2007/O124149 A1 5, 2007 Shen et al. 2007/0255979 A1 1 1/2007 Deily et al. 2007/O124676 A1 5/2007 Amundsen et al. 2007/0257890 A1 1 1/2007 Hotelling et al. 2007/0127888 A1 6/2007 Hayashi et al. 2007,0258642 A1 11/2007 Thota 2007/O128777 A1 6, 2007 Yin et al. 2007/0260460 A1 1 1/2007 Hyatt 2007/0129059 A1 6/2007 Nadarajah et al. 2007/0260595 A1 1 1/2007 Beatty et al. 2007/0130014 A1 6/2007 Altberg et al. 2007/0260822. A 1 1/2007 Adams 2007/013.0128 A1 6/2007 Garg et al. 2007/0261080 Al 11/2007 Saetti 2007, 0132738 A1 6, 2007 Lowles et al. 2007,0265831 A1 11/2007 Dinur et al. 2007/0135949 A1 6, 2007 Snover et al. 2007/0271 104 Al 11/2007 Mckay 2007. O136064 A1 6, 2007 Carroll 2007/0271510 A1 11/2007 Grigoriu et al. 2007/0136778 A1 6/2007 Birger et al. 2007/0274468 Al 11/2007 Cai 2007.0143163 A1 6, 2007 Weiss et al. 2007/0276651 A1 11/2007 Bliss et al. 2007. O149252 A1 6, 2007 JobS et al. 2007/0276714 A1 1 1/2007 Beringer 2007. O150842 A1 6, 2007 Chaudhri et al. 2007/0276810 Al 11/2007 Rosen 2007. O152978 A1 7, 2007 Kocenda et al. 2007/0277088 Al 11/2007 Bodin et al. 2007/0152980 A1 7/2007 Kocienda et al. 2007/0282595 Al 12/2007 Tunning et al. 2007/O155346 A1 7/2007 Mijatovic et al. 2007/0285958 A1 12/2007 Patchta et al. 2007. O15641.0 A1 7, 2007 Stohr et al. 2007/0286363 A1 12/2007 Burg et al. 2007/O156627 A1 7/2007 DAlicandro 2007/0288241 A1 12/2007 Cross et al. 2007. O157089 A1 7/2007 Van OS et al. 2007/0288449 A1 12/2007 Datta et al. 2007/O157268 A1 7/2007 Girish et al. 2007,0291 108 A1 12, 2007 Huber et al. 2007/0162296 A1 7/2007 Altberg et al. 2007/0294.077 Al 12/2007 Narayanan et al. 2007/0162414 A1 7/2007 Horowitz et al. 2007/0294.263 A1 12/2007 Punjet al. 2007/0168922 A1 7/2007 Kaiser et al. 2007,02996.64 A1 12/2007 Peters et al. 2007/0173233 A1 7/2007 Vander Veen et al. 2007/0299831 A1 12/2007 Williams et al. 2007/0173267 A1 7, 2007 Klassen et al. 2007/0300140 Al 12/2007 Makela et al. 2007/0174188 A1 T/2007 Fish 2008, OO 10355 A1 1/2008 Vieri et al. 2007/0174396 A1 7, 2007 Kumar et al. 2008/001295.0 A1 1/2008 Lee et al. 2007/0179776 A1 8/2007 Segond et al. 2008, OO13751 A1 1/2008 HiSelius 2007/0179778 A1 8/2007 Gong et al. 2008.0015864 A1 1/2008 ROSS et al...... 704/275 2007,0180383 A1 8, 2007 Naik 2008, OO16575 A1 1/2008 Vincent et al. 2007/0182595 A1 8, 2007 Ghasabian 2008, 0021708 A1 1/2008 Bennett et al. 2007,0185.551 A1 8, 2007 Meadows et al. 2008/0022208 A1 1/2008 Morse 2007,0185754 A1 8, 2007 Schmidt 2008.0031475 A1 2/2008 Goldstein 2007/O185831 A1 8, 2007 Churcher 2008.0034032 A1 2/2008 Healey et al. 2007,0185917 A1 8, 2007 Prahladet al. 2008.0034044 A1 2/2008 Bhakta et al. 2007,0188901 A1 8, 2007 Heckerman et al. 2008.0036743 A1 2/2008 Westerman et al. 2007,0192027 A1 8, 2007 Lee et al. 2008/0040339 A1 2/2008 Zhou et al. 2007/0192105 A1 8/2007 Neeracher et al. 2008/0042970 A 2.2008 Liang et al. 2007/0192293 A1 8, 2007 Swen 2008.0043936 A1 2/2008 Liebermann 2007,0192403 A1 8, 2007 Heine et al. 2008/0043943 A1 2/2008 Sipher et al. 2007/0192744 A1 8/2007 Reponen 2008, 0046239 A1 2/2008 Boo 2007/O198269 A1 8, 2007 Braho et al. 2008, 0046422 A1 2/2008 Lee et al. 2007. O198273 A1 8, 2007 Hennecke 2008, 0046820 A1 2/2008 Lee et al. 2007/O198566 A1 8, 2007 SuStik 2008, 0046948 A1 2/2008 VeroSub 2007/0203955 A1 8, 2007 Pomerantz 2008/00489.08 A1 2/2008 Sato 2007/0207785 A1 9/2007 Chatterjee et al. 2008/0052063 A1 2/2008 Bennett et al. 2007/0208555 A1* 9, 2007 Blass ...... TO4/9 2008.OO52O73 A1 2/2008 Goto et al. 2007/0208569 A1 9, 2007 Subramanian et al. 2008/0052077 A1 2/2008 Bennett et al. 2007/02O8579 A1 9, 2007 Peterson 2008.0056459 A1 3/2008 Vallier et al. 2007/0208726 A1 9/2007 Krishnaprasad et al. 2008.0056579 A1 3, 2008 Guha 2007/0211071 A1 9, 2007 Slotznicket al. 2008/0057922 A1 3, 2008 Kokes et al. 2007/0213099 A1 9, 2007 Bast 2008/005919.0 A1 3/2008 Chu et al. 2007/0213857 A1 9, 2007 Bodin et al. 2008/0059200 A1 3, 2008 Puli 2007/0213984 A1* 9/2007 Ativanichayaphong G06F 3/167 2008/0059876 A1 3, 2008 Hantler et al. 2008.0062141 A1 3/2008 Chandhri 704/257 2008, OO65382 A1 3, 2008 Ger1 et al. 2007/0219777 A1 9, 2007 Chu et al. 2008, OO65387 A1 3, 2008 CrOSS et al. 2007/0219803 A1 9, 2007 Chiu et al. 2008, 0071529 A1 3/2008 Silverman et al. 2007/0219983 A1 9, 2007 Fish 2008/007 1544 A1 3/2008 Beaufays et al. 2007/0225980 A1 9, 2007 Sumita 2008.0075296 A1 3, 2008 Lindahl et al. 2007/0225984 A1 9, 2007 Milstein et al. 2008/0076972 A1 3/2008 Dorogusker et al. 2007/0226652 A1 9, 2007 Kikuchi et al. 2008.OO7731.0 A1 3/2008 Murlidar et al. US 9,318.108 B2 Page20

(56) References Cited 2008/018960.6 A1 8/2008 Rybak 2008. O195312 A1 8/2008 Aaron et al. U.S. PATENT DOCUMENTS 2008. O195601 A1 8, 2008 Ntoulas et al. 2008. O195940 A1 8, 2008 Gail et al. 2008.OO77384 A1 3/2008 Agapi et al. 2008, 0200142 A1 8, 2008 Abdel-Kader et al. 2008.OO77386 A1 3/2008 Gao et al. 2008/02O1306 A1 8/2008 Cooper et al. 2008.OO77391 A1 3/2008 Chino et al. 2008/02O1375 A1 8, 2008 Khedouri et al. 2008.OO77393 A1 3/2008 Gao et al. 2008/0204379 A1 8/2008 Perez-Noguera 2008.0077406 A1 3/2008 Ganong, III 2008/0207176 A1 8, 2008 Brackbill et al. 2008.OO77859 A1 3/2008 Schabes et al. 2008, 0208585 A1 8/2008 Ativanichayaphong et al. 2008.OO79566 A1 4/2008 Singh et al. 2008/02O8587 A1 8, 2008 Ben-david et al. 2008/0082332 A1 4/2008 Mallett et al. 2008, 0212796 A1 9, 2008 Denda 2008/0082338 A1 4/2008 O'Neil et al. 2008/0219641 A1 9, 2008 Sandrew et al. 2008/0082390 A1 4/2008 Hawkins et al. 2008/0221866 A1 9/2008 Katragadda et al. 2008.OO82576 A1 4/2008 Bodin et al. 2008/0221879 A1 9, 2008 Cerra et al. 2008, OO82651 A1 4/2008 Singh et al. 2008, 0221880 A1 9, 2008 Cerra et al. 2008, OO84974 A1 4/2008 Dhanakshirur 2008, 0221889 A1 9, 2008 Cerra et al. 2008/009 1406 A1 4/2008 Baldwin et al. 2008/0221903 Al 9, 2008 Kanevsky et al. 2008/0091426 A1 4/2008 Rempel et al. 2008/02221 18 Al 9, 2008 Scian et al. 2008.009 1443 A1 4/2008 Strope et al. 2008/0228463 A1 9, 2008 Mori et al. 2008/0096531 A1 4/2008 Mcquaide et al. 2008/0228485 Al 9, 2008 Owen 2008, 0096726 A1 4/2008 Riley et al. 2008/0228490 A1 9, 2008 Fischer et al. 2008/0097937 A1 4/2008 Hadjarian 2008/0228495 A1 9, 2008 Cross et al. 2008/00983O2 A1 4/2008 Roose 2008/0228496 A1 9, 2008 Yu et al. 2008/0098480 A1 4/2008 Henry et al. 2008/0228928 A1 9, 2008 Donelli et al. 2008/0100579 A1 5/2008 Robinson et al. 2008/0229185 A1 9/2008 Lynch 2008/0101584 A1 5/2008 Gray et al. 2008/0229218 A1 9/2008 Maeng 2008.0109222 A1 5, 2008 Liu 2008, 0235024 A1 9/2008 Goldberg et al. 2008/0109402 A1 5/2008 Wang et al. 2008/0235027 Al 9, 2008 Cross 2008.0114480 A1 5, 2008 Harb 2008, 0240569 A1 10, 2008 Tonouchi 2008.0114598 A1 5, 2008 Prieto et al. 2008/0242280 A1 10/2008 Shapiro et al. 2008.0114841 A1 5, 2008 Lambert 2008/0244390 A1 10, 2008 Fux et al. 2008.0118143 A1 5, 2008 Gordon et al. 2008/024444.6 A1 10, 2008 Lefevre et al. 2008/O12O102 A1 5, 2008 Rao 2008, O247519 A1 10, 2008 Abella et al. 2008/O12O112 A1 5, 2008 Jordan et al. 2008/0248797 A1 10, 2008 Freeman et al. 2008/O120342 A1 5/2008 Reed et al. 2008/0249770 Al 10/2008 Kim et al. 2008.0122796 A1 5/2008 Jobs et al. 2008/0253577 A1 10/2008 Eppolito 2008/O126O77 A1 5, 2008 Thorn 2008, 0255837 A1 10, 2008 Kahn et al. 2008/O126091 A1 5, 2008 Clark et al. 2008, 0255845 A1 10, 2008 Bennett 2008/O126093 A1 5, 2008 Sivadas 2008, O256613 A1 10, 2008 Grover 2008/O126100 A1 5, 2008 Grost et al. 2008/0259022 A1 10, 2008 Mansfield et al. 2008/0129520 A1 6, 2008 Lee 2008/0262838 A1 10, 2008 Nurminen et al. 2008. O130867 A1 6, 2008 Bowen 2008/0262846 A1 10, 2008 Burns et al. 2008. O131006 A1 6, 2008 Oliver 2008/02701 18 A1 10, 2008 Kuo et al. 2008/0132221 A1 6/2008 Willey et al. 2008/0270.138 Al 10/2008 Knight et al. 2008. O133215 A1 6, 2008 Sarukkai 2008/0270.139 A1 10, 2008 Shi et al. 2008/033228 A1 6, 2008 Rao 2008/0270.140 A1 10, 2008 Hertz et al. 2008. O133241 A1 6, 2008 Baker et al. 2008/0277473 Al 1 1/2008 Kotlarsky et al. 2008. O133956 A1 6, 2008 Fadell 2008/028151.0 A1 11/2008 Shahine 2008. O140413 A1 6, 2008 Millman et al. 2008/02921 12 A1 11/2008 Valenzuela et al. 2008. O140416 A1 6, 2008 Shostalk 2008/0294418 A1 1 1/2008 Cleary et al. 2008/O140652 A1 6/2008 Millman et al. 2008/0294651 A1 1 1/2008 Masuyama et al. 2008. O140657 A1 6, 2008 AZvine et al. 2008/0294981 A1 11/2008 Balzano et al. 2008/O141125 A1 6/2008 Ghassabian et al. 2008/0298766 Al 12/2008 Wen et al. 2008. O141180 A1 6, 2008 Reed et al. 2008, 0299523 A1 12/2008 Chai et al. 2008/O141182 A1 6/2008 Barsness et al. 2008/0300871 A1 12/2008 Gilbert 2008. O146245 A1 6/2008 Appaji 2008/0300878 A1 12/2008 Bennett 2008. O146290 A1 6, 2008 Sreeram et al. 2008/0306727 A1 12/2008 Thurmair et al. 2008. O147408 A1 6, 2008 Da Palma et al. 2008/0312909 A1 12/2008 Hermansen et al. 2008/O1474 11 A1 6/2008 Dames et al. 2008/0313335 Al 12/2008 Jung et al. 2008. O1478.74 A1 6, 2008 Yoneda et al. 2008/031.6183 Al 12/2008 Westerman et al. 2008. O150900 A1 6, 2008 Han 2008/0319753 A1 12/2008 Hancock 2008. O154600 A1 6, 2008 Tian et al. 2008/0319763 A1 12/2008 Di Fabbrizio et al. 2008. O1546 12 A1 6/2008 Evermann et al. 2009,0003115 A1 1/2009 Lindahl et al. 2008. O154828 A1 6/2008 Antebi et al. 2009/0005012 A1 1/2009 Van Heugten 2008. O157867 A1 7, 2008 Krah 2009, OOO5891 A1 1/2009 Batson et al. 2008. O163119 A1 7/2008 Kim et al. 2009, OOO6097 A1 1/2009 Etezadi et al. 2008. O163131 A1 7, 2008 Hirai et al. 2009/0006099 A1 1/2009 Sharpe et al. 2008/O165144 A1 7/2008 Forstall et al. 2009/0006100 A1* 1/2009 Badger ...... 704/275 2008. O165980 A1 7, 2008 Pavlovic et al. 2009,0006343 A1 1/2009 Platt et al. 2008. O165994 A1 7/2008 Caren et al. 2009,0006345 A1 1/2009 Platt et al. 2008.0167013 A1 7/2008 Novicket al. 2009,0006488 A1 1/2009 Lindahl et al. 2008.0167858 A1 7/2008 Christie et al. 2009 OOO6671 A1 1/2009 Batson et al. 2008. O168366 A1 7/2008 Kocienda et al. 2009, OOOTOO1 A1 1/2009 Morin et al. 2008. O183473 A1 7/2008 Nagano et al. 2009, OO 11709 A1 1/2009 Akasaka et al. 2008. O189099 A1 8, 2008 Friedman et al. 2009 OO12748 A1 1/2009 Beish et al. 2008. O1891.06 A1 8, 2008 Low et al. 2009, OO12775 A1 1/2009 El Hady et al. 2008. O1891.1.0 A1 8, 2008 Freeman et al. 2009 OO18828 A1 1/2009 Nakadai et al. 2008. O1891.14 A1 8, 2008 Fail et al. 2009 OO18834 A1 1/2009 Cooper et al. US 9,318.108 B2 Page 21

(56) References Cited 2009.0167509 A1 7, 2009 Fadell et al. 2009/0171578 A1 7/2009 Kim et al. U.S. PATENT DOCUMENTS 2009.0171664 A1 7/2009 Kennewicket al. 2009,0172108 A1 7/2009 Singh 2009 OO18835 A1 1/2009 Cooper et al. 2009/0172542 A1 7/2009 Girish et al. 2009 OO18839 A1 1/2009 Cooper et al. 2009/O174667 A1 7/2009 KOcienda et al. 2009 OO1884.0 A1 1/2009 Lutz et al. 2009/017.4677 A1 7/2009 Gehani et al. 2009/0022329 A1 1/2009 Mahowald 2009/0177300 Al 72009 Lee 2009/0028435 A1 1/2009 Wu et al. 2009/0177461 A1 7/2009 Ehsani et al. 2009,003 0800 A1 1/2009 Grois 2009,0182445 A1 7/2009 Girish et al. 20090030978 A1 1/2009 Johnson et al. 2009/01874.02 Al 72009 Scholl 2009.0043583 A1 2/2009 Agapi et al. 2009/O187577 A1 7/2009 Reznik et al. 2009/004.8821 A1 2/2009 Yam et al. 2009,019 1895 A1 7/2009 Singh et al. 2009/0048845 A1 2/2009 Burckart et al. 2009,0192782 A1 7, 2009 Drewes 2009.0049067 A1 2/2009 Murray 2009/O198497 A1 8, 2009 Kwon 2009/0055.179 A1 2/2009 Cho et al. 2009/0204409 A1 8, 2009 Mozer et al. 2009/0055186 A1 2/2009 Lance et al. 2009, 0210232 A1 8, 2009 Sanchez et al. 2009 OO58823 A1 3/2009 Kocienda 2009, 0213134 A1 8/2009 Stephanicket al. 2009 OO5886.0 A1 3/2009 Fong et al. 2009/0215503 A1 8/2009 Zhang et al. 2009 OO60472 A1 3, 2009 Bull et al. 2009/0216704 All 82009 Zheng et al. 2009, OO63974 A1 3/2009 Bullet al. 2009/0222270 A2 9, 2009 Likens et al. 2009 OO64031 A1 3/2009 Bullet al. 2009, 0222488 A1 9, 2009 Boerries et al. 2009/007OO97 A1 3, 2009 Wu et al. 2009/0228126 A1 9/2009 Spielberg et al. 2009/0070102 A1 3/2009 Maegawa 2009/0228273 Al 9, 2009 Wang et al. 2009 OO701 14 A1 3/2009 Staszak 2009/0228281 A1 9/2009 Singleton et al. 2009 OO74214 A1 3, 2009 Bradford et al. 2009/0228792 A1 9, 2009 Van OS et al. 2009/0076792 A1 3/2009 Lawson-Tancred 2009/0228842 A1 9, 2009 Westerman et al. 2009/0076796 A1 3, 2009 Daraselia 2009, 0234655 A1 9, 2009 Kwon 2009/0076819 A1 3, 2009 Wouters et al. 2009/0239202 Al 9, 2009 Stone 2009/0076821 A1 3/2009 Brenner et al. 2009/0239552 A1 9, 2009 Churchill et al. 2009/0076825 A1 3, 2009 Bradford et al. 2009/0240485 Al 9, 2009 Dalal et al. 2009, OO77165 A1 3/2009 Rhodes et al. 2009,0241054 A1 9, 2009 Hendricks 2009/0083.034 A1 3f2009 Hernandez et al. 2009/024.1760 Al 10/2009 Georges 2009,0083.035 A1 3/2009 Huang et al. 2009/0247237 A1 10, 2009 Mittleman et al. 2009,0083.036 A1 3, 2009 Zhao et al. 2009/0248.182 Al 10/2009 Logan et al. 2009/008.3037 A1 3f2009 Gleason et al. 2009/0248420 Al 10/2009 Basir et al. 2009,008.3047 A1 3f2009 Lindahl et al. 2009/0249198 Al 10/2009 Davis et al. 2009/0089058 A1 4/2009 Bellegarda 2009/0252350 Al 10/2009 Seguin 2009/0092260 A1 4, 2009 Powers 2009/0253457 Al 10/2009 Seguin 2009/0092261 A1 4, 2009 Bard 2009,0253463 A1 10, 2009 Shin et al. 2009/0092262 A1 4/2009 Costa et al. 2009/0254339 A1 10/2009 Seguin 2009,009.4029 A1 4/2009 Koch et al. 2009,0254345 A1 10, 2009 Fleizach et al. 2009/0094033 A1 4/2009 Mozer et al. 2009,0259969 A1 10, 2009 Pallakoff 2009, OO97634 A1 4/2009 Nambiar et al. 2009,0265368 A1* 10, 2009 Crider et al...... 707/102 2009/0097637 A1 4, 2009 Boscher et al. 2009/0271 109 A1 10/2009 Lee et al. 2009,010.0049 A1 4/2009 Cao 2009,0271175 A1 10, 2009 Bodin et al. 2009,010.0454 A1 4, 2009 Weber 2009,0271176 A1 10, 2009 Bodin et al. 2009, O104898 A1 4/2009 Harris 2009,0271178 A1 10, 2009 Bodin et al. 2009/0106026 A1 4/2009 Ferrieux 2009/02743 15 A1 11/2009 Carnes et al. 2009/0106376 A1 4/2009 Tom et al. 2009,0281789 A1 11/2009 Waibel et al. 2009/0106397 A1 4, 2009 O'Keefe 2009/0284482 Al 11/2009 Chin 2009/O112572 A1 4/2009 Thorn 2009/0286,514 A1 11/2009 Lichorowic et al. 2009/O112677 A1 4, 2009 Rhett 2009,0287583 A1 11/2009 Holmes 2009/01 12892 A1 4/2009 Cardie et al. 2009, 0290718 A1 11/2009 Kahn et al. 2009/01 19587 A1 5, 2009 Allen et al. 2009,0292987 A1 11/2009 Sorenson 2009, O123021 A1 5/2009 Jung et al. 2009,0296.552 A1 12/2009 Hicks et al. 2009, O123071 A1 5, 2009. Iwasaki 2009,0298.474 Al 12/2009 George 2009/O125477 A1 5, 2009 Lu et al. 2009, 0299.745 A1 12/2009 Kennewicket al. 2009/0128505 A1 5/2009 Partridge et al. 2009, 0299849 A1 12/2009 Cao et al. 2009.0137286 A1 5, 2009 Luke et al. 2009/0300391 Al 12/2009 Jessup et al. 2009, O138736 A1 5, 2009 Chin 2009/0300488 A1 12/2009 Salamon et al. 2009, O138828 A1 5, 2009 Schultz et al. 20090304.198 A1 12/2009 Herre et al. 2009, O144049 A1 6, 2009 Haddad et al. 2009,0306967 A1 12/2009 Nicolov et al. 2009. O144428 A1 6/2009 Bowater et al. 20090306969 Al 12/2009 Goud et al. 2009, O144609 A1 6/2009 Liang et al. 2009/0306979 A1 12/2009 Jaiswal et al. 2009/0146848 A1 6/2009 Ghassabian 2009,0306980 A1 12/2009 Shin 2009. O15O147 A1 6/2009 Jacoby et al. 2009/0306981 A1 12/2009 Cromacket al. 2009. O15O156 A1 6, 2009 Kennewicket al. 2009,0306985 A1 12/2009 Roberts et al. 2009, O153288 A1 6/2009 Hope et al. 2009,0306988 A1 12/2009 Chen et al. 2009, O154669 A1 6, 2009 Wood et al. 2009/0306989 A1 12/2009 Kaji 2009. O157382 A1 6, 2009 Bar 2009/0307162 A1 12/2009 Bui et al. 2009. O157384 A1 6/2009 Toutanova et al. 2009/0307201 Al 12/2009 Dunning et al. 2009. O1574O1 A1 6, 2009 Bennett 2009/0307584 A1 12/2009 Davidson et al. 2009, O158423 A1 6, 2009 Orlassino et al. 20090313023 A1 12/2009 Jones 2009.0160803 A1 6, 2009 Hashimoto 2009/0313026 A1 12/2009 Coffman et al. 2009, O164441 A1 6/2009 Cheyer 20090313544 A1 12/2009 Wood et al. 2009, O164655 A1 6, 2009 Pettersson et al. 20090313564 A1 12/2009 Rottler et al. 2009, O164937 A1 6, 2009 Alviar et al. 2009/0316943 Al 12/2009 Frigola Munoz et al. 2009.0167508 A1 7/2009 Fadell et al. 20090318119 A1 12/2009 Basir et al. US 9,318.108 B2 Page 22

(56) References Cited 2010, 0169075 A1 7, 2010 Raffa et al. 2010, 0169097 A1 7, 2010 Nachman et al. U.S. PATENT DOCUMENTS 2010.0171713 A1 7, 2010 Kwok et al. 2010/0174544 A1 7, 2010 Heifets 20090318198 A1 12, 2009 Carroll 2010/0179932 A1 7/2010 Yoon et al. 20090319266 A1 12, 2009 Brown et al. 2010/0179991 Al 72010 Lorchet al. 2009/0326936 Al 12/2009 Nagashima 2010.0185448 A1 7, 2010 Meisel 2009,032.6938 A1 12, 2009 Maria et al. 2010/0185949 A1 7/2010 Jaeger 2009/0326949 A1 12, 2009 Douthitt et al. 2010, O197359 A1 8, 2010 Harris 2009/0327977 A1 12, 2009 Bachfischer et al. 2010.01992 15 A1 8/2010 Seymour et al. 2010, 0004931 A1 1/2010 Ma et al. 2010/0204986 A1 8/2010 Kennewicket al. 2010.0005081 A1 1/2010 Bennett 2010, 0211199 A1 8, 2010 Naik et al. 2010, 0013796 A1 1/2010 Abileah et al. 2010, 0217604 A1 8, 2010 Baldwin et al. 2010.00 19834 A1 1/2010 Zerbe et al. 2010/0222033 A1 9, 2010 Scott et al. 2010, 0023318 A1 1/2010 Lemoine 2010/0222098 A1 9/2010 Garg 2010, 0023320 A1 1/2010 Di Cristo et al. 2010/0223055 A1 9, 2010 Mclean 2010/0030928 A1 2/2010 Conroy et al. 2010/02231.31 A1 9, 2010 Scott et al. 2010.0031143 A1 2/2010 Rao et al. 2010/0228.540 A1 9, 2010 Bennett 2010.0036655 A1 2/2010 Cecil et al. 2010/0228691 Al 9, 2010 Yang et al. 2010.0036660 A1 2/2010 Bennett 2010. 0231474 A1 9/2010 Yamagajo et al. 2010/0037183 A1 2/2010 Miyashita et al. 2010. 0235167 A1 9, 2010 Bourdon 2010.0042400 A1 2/2010 Blocket al. 2010/0235341 Al 9, 2010 Bennett 2010.0049514 A1 2/2010 Kennewicket al. 2010/0235729 A1 9/2010 Kocienda et al. 2010.005OO64 A1 2/2010 Liu et al. 2010/023577O A1 9/2010 Ording et al. 2010.0054512 A1 3, 2010 Solum 2010/0250542 Al 9, 2010 Fujimaki 2010.0057457 A1 3/2010 Ogata et al. 2010, O250599 A1 9/2010 Schmidt et al. 2010.0057643 A1 3/2010 Yang 2010, O257160 A1 10, 2010 Cao 2010.006.0646 A1 3/2010 Unsal et al. 2010/0257478 Al 10/2010 Longe et al. 2010.0063804 A1 3/2010 Sato et al. 2010, O262599 A1 10, 2010 Nitz 2010.0063825 A1 3/2010 Williams et al. 2010/0268539 Al 10/2010 Xu et al. 2010.0063961 A1 3/2010 Guilheneufetal. 2010/0274753 A1 10/2010 Liberty et al. 2010.00641 13 A1 3/2010 Lindahl et al. 2010/0277579 A1 11, 2010 Cho et al. 2010, OO67723 A1 3/2010 Bergmann et al. 2010/0278320 A1 11, 2010 Arsenault et al. 2010 OO67867 A1 3, 2010 Lin et al. 2010/0278453 A1 1 1/2010 King 2010, 0070281 A1 3/2010 Conkie et al. 2010/0280983 A1 11, 2010 Cho et al. 2010.0070899 A1 3/2010 Hunt et al. 2010/0281 034 A1 11/2010 Petrou et al. 2010.0076760 A1 3/2010 Kraenzel et al. 2010/0286985 A1 11/2010 Kennewicket al. 2010, 00795O1 A1 4/2010 Ikeda et al. 2010/0287514 A1 1 1/2010 Cragun et al. 2010, 0080398 A1 4/2010 Waldmann 2010, O293460 A1 11, 2010 Budelli 2010, 0080470 A1 4/2010 Deluca et al. 2010/02991.33 Al 11/2010 Kopparapu et al. 2010/0081456 A1 4/2010 Singh et al. 2010, O299138 A1 11, 2010 Kim 2010.0081487 A1 4/2010 Chen et al. 2010/0299142 A1 11/2010 Freeman et al. 2010.0082327 A1 4/2010 Rogers et al. 2010/0302056 All 12/2010 Dutton et al. 2010.0082328 A1 4/2010 Rogers et al. 2010/0305807 A1 12/2010 Basir et al. 2010.0082329 A1 4/2010 Silverman et al. 2010/0305947 A1 12/2010 Schwarz et al. 2010, OO82346 A1 4/2010 Rogers et al. 2010.0312547 A1* 12/2010 Van OS ...... GO6F 3/167 2010/0082347 A1 4/2010 Rogers et al. TO4/9 2010.0082348 A1 4/2010 Silverman et al. 2010/0312566 Al 12/2010 Odinak et al. 2010/0082349 A1 4/2010 Bellegarda et al. 2010/0318576 Al 12/2010 Kim 2010.0082970 A1 4/2010 Lindahl et al. 2010/0322438 Al 12/2010 Siotis 2010.0086152 A1 4/2010 Rank et al. 2010/0324895 A1 12/2010 Kurzweil et al. 2010.0086153 A1 4/2010 Hagen et al. 2010/0324905 A1 12/2010 Kurzweil et al. 2010.0086156 A1 4/2010 Rank et al. 2010/0325573 A1 12/2010 EStrada et al. 2010, OO88020 A1 4/2010 Sano et al. 2010/0325588 Al 12/2010 Reddy et al. 2010, OO88093 A1 4/2010 Lee et al. 2010/0332224 A1 12/2010 Makela et al. 2010, OO88100 A1 4/2010 Lindahl 2010/0332235 A1 12/2010 David 2010, 01 00212 A1 4/2010 Lindahl et al. 2010/0332280 Al 12/2010 Bradley et al. 2010, 0100384 A1 4/2010 Ju et al. 2010/0332348 A1 12/2010 Cao 2010.01.03776 A1 4/2010 Chan 2010/0332428 A1 12/2010 Mchenry et al. 2010/0106500 A1 4/2010 Mckee et al. 2010/0332976 A1 12/2010 Fux et al. 2010/01 1485.6 A1 5/2010 Kuboyama 2010/033303.0 A1 12/2010 Johns 2010, 0125460 A1 5, 2010 Mellottet al. 2011/0002487 A1 1/2011 Panther et al. 2010.0125811 A1 5, 2010 Moore et al. 2011/0010178 A1 1/2011 Lee et al. 2010, 0131273 A1 5/2010 Aley-Raz et al. 2011 OO 10644 A1 1/2011 Merrill et al...... 715,762 2010, 0131498 A1 5, 2010 Linthicum et al. 2011/0016150 A1 1/2011 Engstrom et al. 2010, 0131899 A1 5, 2010 Hubert 2011 OO18695 A1 1/2011 Bells et al. 2010, 0138215 A1 6, 2010 Williams 2011, 0021213 A1 1/2011 Carr 2010/01382.24 A1 6/2010 Bedingfield, Sr. 2011/0022292 A 1 1/2011 Shen et al. 2010, 0138416 A1 6, 2010 Bellotti 2011/0022394 A1 1/2011 Wide et al. 2010/0138759 A1* 6/2010 Roy ...... G06F 9/4443 2011/0022952 A1 1/2011 Wu et al. 715,764 2011/00296.16 A1 2/2011 Wang et al. 2010/0142740 A1 6/2010 Roerup 2011/0033064 A1 2/2011 Johnson et al. 2010, 0145694 A1 6, 2010 Ju et al. 2011/0038489 A1 2/2011 Visser et al. 2010, 0145700 A1 6, 2010 Kennewicket al. 2011/0047072 A1 2/2011 Curea 2010/0146442 A1 6/2010 Nagasaka et al. 2011/0047161 A1 2/2011 Myaeng et al. 2010.01531 15 A1 6, 2010 Klee et al. 2011/0050591 A1 3, 2011 Kim et al. 2010, 0161313 A1 6, 2010 Karttunen 2011/0054901 A1 3/2011 Qin et al. 2010, 0161554 A1 6, 2010 Datuashvili et al. 2011/0055256 A1 3/2011 Phillips et al. 2010, 0164897 A1 7, 2010 Morin et al. 2011/0060584 A1 3/2011 Ferrucci et al. US 9,318.108 B2 Page 23

(56) References Cited 2011/0314404 A1 12/2011 Kotler et al. 2012,0002820 A1 1/2012 Leichter U.S. PATENT DOCUMENTS 2012/0011138 A1 1/2012 Dunning et al. 2012fOO13609 A1 1/2012 Reponen et al. 2011/0060587 A1 3/2011 Phillips et al. 2012/006678 A 1 1/2012 Gruber et al. 2011/0060807 A1 3f2011 Martin et al. 2012/0020490 A1 1/2012 Leichter 2011/0066468 A1 3/2011 Huang et al. 2012/0022787 A1 1/2012 LeBeau et al. 2011 OO72492 A1 3/2011 Mohler et al. 2012/0022857 A1 1/2012 Baldwin et al. 2011/0076994 A1 3, 2011 Kim et al. 2012/0022860 A1 1/2012 Lloyd et al. 2011 OO82688 A1 4/2011 Kim et al. 2012/0022868 A1 1/2012 LeBeau et al. 2011, 0083079 A1 4, 2011 Farrell et al. 2012/0022869 A1 1/2012 Lloyd et al. 2011/0087491 A1 4/2011 Wittenstein et al. 2012/0022870 A1 1/2012 Kristjansson et al. 2011/0090078 A1 4/2011 Kim et al. 2012/0022872 A1 1/2012 Gruber et al. 2011/0093261 A1 4/2011 Angott 2012/0022874 A1 1/2012 Lloyd et al. 2011/0093265 A1 4/2011 Stent et al. 2012/0022876 A1 1/2012 LeBeau et al. 2011/00932.71 A1 4/2011 Bernard et al. 2012/0023088 A1 1/2012 Cheng et al. 2011/0099000 A1 4/2011 Rai et al. 2012fOO34904 A1 2/2012 LeBeau et al. 2011/0103682 A1 5, 2011 Chidlovskii et al. 2012/0035907 A1 2/2012 Lebeau et al. 2011/0106736 A1 5/2011 Aharonson et al. 2012,003.5908 A1 2/2012 LeBeau et al. 2011 01105O2 A1 5/2011 Daye et al. 2012, 0035924 A1 2/2012 Jitkoff et al. 2011/01 12827 A1 5, 2011 Kennewicket al. 2012,003.5925 A1 2/2012 Friend et al. 2011/01 12837 A1 5, 2011 Kurki-Suonio et al. 2012/0035931 A1 2/2012 LeBeau et al. 2011/01 12921 A1 5, 2011 Kennewicket al. 2012, 0035932 A1 2/2012 Jitkoff et al. 2011 011661.0 A1 5, 2011 Shaw et al. 2012.0036556 A1 2/2012 LeBeau et al. 2011 0119049 A1 5, 2011 Y1onen 2012/0042343 A1 2/2012 Laligand et al. 2011 0119051 A1 5, 2011 Li et al. 2012.0053815 A1 3/2012 Montanarietal. 2011/O125540 A1 5/2011 Jang et al. 2012.0053945 A1 3/2012 Gupta et al. 2011/O130958 A1 6, 2011 Stahl et al. 2012.0056815 A1 3, 2012 Mehra 2011/013 103.6 A1 6, 2011 Di Cristo et al. 2012/0078627 A1 3/2012 Wagner 2011/013 1038 A1 6/2011 Oyaizu et al. 2012,00823 17 Al 4, 2012 Pance et al. 2011 0131045 A1 6, 2011 Cristo et al. 2012fOO84086 A1 4/2012 Gilbert et al. 2011/0143811 A1 6/2011 Rodriguez 2012/0108221 A1 5/2012 Thomas et al. 2011/O144973 Al 6, 2011 Bocchieri et al. 2012/0116770 A1 5, 2012 Chen et al. 2011/O144999 A1 6/2011 Jang et al. 2012/O124126 A1 5/2012 Alcazar et al. 2011/O145718 A1 6, 2011 Ketola et al. 2012/0136572 A1 5, 2012 Norton 2011/015 1830 A1 6, 2011 Blanda et al. 2012/0137367 A1 5/2012 Dupont et al. 2011 O153209 A1 6/2011 Geelen 2012O149394 A1 6/2012 Singh et al. 2011/O153330 A1 6, 2011 Yazdani et al. 2012. O150580 A1 6, 2012 Norton 2011/0153373 A1 6/2011 Dantzig et al. 2012/0158293 A1 6, 2012 Burnham 2011/O157029 A1 6/2011 Tseng 2012/0158422 A1 6/2012 Burnham et al. 2011/O161076 A1 6, 2011 Davis et al. 2012fO16371.0 A1 6, 2012 Skaffet al. 2011/O161079 A1 6, 2011 Gruhn et al. 2012/0173464 A1 7/2012 Tur et al. 2011/0161309 A1 6/2011 Lung et al. 2012/0174121 A1 7/2012 Treat et al. 2011/0161852 A1 6, 2011 Vainio et al. 2012/0185237 A1 7/2012 Gajic et al. 2011/O167350 A1 7, 2011 Hoellwarth 2012fO197995 A1 8, 2012 Caruso 2011/017581.0 A1 7, 2011 Markovic et al. 2012/0197998 A1 8/2012 Kessel et al. 2011/0179002 A1 7, 2011 Dumitru et al. 2012fO2O1362 A1 8/2012 Crossan et al. 2011/0179372 A1 7, 2011 Moore et al. 2012/0214141 A1 8/2012 Raya et al. 2011/O184721 A1 7, 2011 Subramanian et al. 2012/0214517 A1 8/2012 Singh et al. 2011/O184730 A1 7, 2011 LeBeau et al. 2012/0221339 A1 8/2012 Wang et al. 2011 0191271 A1 8, 2011 Baker et al. 2012fO221552 A1 8/2012 Reponen et al. 2011 0191344 A1 8, 2011 Jin et al. 2012fO232906 A1 9/2012 Lindahl et al. 2011/O195758 A1 8, 2011 Damale et al. 2012/0245719 A1 9/2012 Story, Jr. et al. 2011/02O1387 A1 8, 2011 Paek et al. 2012/0245941 A1 9/2012 Cheyer 2011/0209088 A1 8/2011 Hinckley et al. 2012/0245944 Al 9, 2012 Gruber et al. 2011/0212717 A1 9, 2011 Rhoads et al. 2012/0252367 A1 10/2012 Gaglio et al. 2011/0218855 A1 9, 2011 Cao et al. 2012fO254152 A1 10, 2012 Park et al. 2011/0219018 A1 9/2011 Bailey et al. 2012/0265528 A1 10, 2012 Gruber et al. 2011/0224972 A1 9, 2011 Millett et al. 2012/0265535 A1 10/2012 Bryant-Rich et al. 2011/0231182 A1 9, 2011 Weider et al. 2012,0271625 A1 10/2012 Bernard 2011/0231188 A1 9, 2011 Kennewicket al. 2012/0271635 A1 10/2012 Ljolje 2011/0231474 A1 9, 2011 Locker et al. 2012/0271676 A1 10, 2012 Aravamudan et al. 2011/0238407 A1 9, 2011 Kent 2012/0284.027 A1 11/2012 Mallett et al. 2011/0238408 A1 9/2011 Larcheveque et al. 2012fO290300 A1 11/2012 Lee et al. 2011/0238676 A1 9, 2011 Liu et al. 2012fO296649 A1 11/2012 Bansal et al. 2011/0242007 A1 10/2011 Gray et al. 2012/0304124 A1 11/2012 Chen et al. 2011/024.9144 A1 10/2011 Chang 2012/0309363 A1 12/2012 Gruber et al. 2011/0260861 A1 10/2011 Singh et al. 2012/03 10642 Al 12/2012 Cao et al. 2011/0264.643 A1 10, 2011 Cao 2012/0310649 A1 12/2012 Cannistraro et al. 2011/0274303 A1 11/2011 Filson et al. 2012,0310652 A1 12/2012 O'Sullivan 2011/0276598 A1 1 1/2011 Kozempel 2012/0311478 A1 12/2012 Van OS et al. 2011/02793.68 A1 11/2011 Klein et al. 2012/0311583 Al 12/2012 Gruber et al. 2011/0282888 A1 1 1/2011 Koperski et al. 2012/0311584 A1 12/2012 Gruber et al. 2011/0288861 A1 11/2011 Kurzweil et al. 2012/0311585 A1 12/2012 Gruber et al. 2011/0298,585 A1 12/2011 Barry 2012/0317498 Al 12/2012 Logan et al. 2011/0306426 A1 12/2011 Novak et al. 2012/0330660 A1 12/2012 Jaiswal 2011/0307491 A1 12, 2011 Fisk et al. 2012/0330661 A1 12/2012 Lindahl 2011/030781.0 A1 12/2011 Hillerio et al. 2013,00054.05 A1 1/2013 Prociw 2011/0314032 A1 12/2011 Bennett et al. 2013/0006633 A1 1/2013 Grokop et al. US 9,318.108 B2 Page 24

(56) References Cited EP O795811 A1 9, 1997 EP O476.972 B1 5, 1998 U.S. PATENT DOCUMENTS EP O845894 A2 6, 1998 EP O863453 A1 9, 1998 2013, OOO6638 A1 1/2013 Lindahl EP O863469 A2 9, 1998 2013,005.5099 A1 2/2013 Yao et al. EP O867860 A2 9, 1998 2013/0073286 A1 3/2013 Bastea-Forte et al. EP O869697 A2 10, 1998 2013,008O167 A1 3, 2013 MOZer EP O88962.6 A1 1/1999 2013,008O177 A1 3, 2013 Chen EP O917O77 A2 5, 1999 2013/0085761 A1 4/2013 Bringert et al. EP 0691023 B1 9, 1999 2013/0110505 A1 5, 2013 Gruber et al. EP O946032 A2 9, 1999 2013/0110515 A1 5, 2013 Guzzoni et al. EP O981.236 A1 2/2000 2013/0110518 A1 5, 2013 Gruber et al. EP O982732 A1 3f2OOO 2013/01 10519 A1 5/2013 Cheyer et al. EP 984430 A2 3, 2000 2013/01 10520 A1 5/2013 Cheyer et al. EP 1001588 A2 5, 2000 2013/011 1348 A1 5, 2013 Gruber et al. EP 1014277 A1 6, 2000 2013/011 1487 A1 5/2013 Cheyer et al. EP 1028425 A2 8, 2000 2013, O115927 A1 5, 2013 Gruber et al. EP 1028426 A2 8, 2000 2013/01 17022 A1 5, 2013 Chen et al. EP 1047251 A2 10, 2000 EP 10763O2 A1 2/2001 2013/0170738 A1 7/2013 Capuozzo et al. EP 109 1615 A1 4/2001 2013,0185074 A1 7, 2013 Gruber et al. EP 1107229 A2 6, 2001

2013,022.5128 A1 8, 2013 Gomar EP 1233600 A2 8, 2002 2013,0238647 A1 9/2013 Thompson EP 1245023 (A1) 10, 2002 2013/0275117 A1 10, 2013 Winer EP 1246,075 A2 10, 2002 2013/0289991 A1 10, 2013 Eshwar et al. EP 1280326 A1 1, 2003 2013/0304758 A1 11/2013 Gruber et al. EP 1311 102 A1 5, 2003 2013/0325443 Al 12/2013 Begeja et al. EP 1315.084 A1 5, 2003 2013/0346068 A1 12/2013 Solem et al. EP 1315086 A1 5, 2003 2014/008.0428 A1 3/2014 Rhoads et al...... 455,88 EP 1347361 A1 9, 2003 2014/0086458 A1 3/2014 Rogers et al. EP 1379061 A2 1/2004 2014/0152577 A1 6, 2014 Yuen et al. EP 1432219 A1 6, 2004 EP 1435620 A1 7, 2004 EP 1480421 A1 11, 2004 FOREIGN PATENT DOCUMENTS EP 1517228 A2 3, 2005 CN 1263385 A 8, 2000 E EA 838. CN 1494695 A 5, 2004 EP 1650938 A1 4, 2006 CN 1673939 A 9, 2005 EP 1693829 A1 8, 2006 CN 1864.204 A 11, 2006 EP 1181802 B1 2/2007 CN 1959628 A S/2007 EP 1818786 A1 8, 2007 CN 101162153. A 4/2008 EP 1892700 A1 2/2008 CN 101.183525 A 5, 2008 EP 1912205 A2 4, 2008 CN 101297541. A 10, 2008 EP 1939860 A1 T 2008 CN 101535983. A 9, 2009 EP O651543 B1 9, 2008 CN 101939.740 A 1, 2011 EP 1909263 B1 1/2009 DE 3837.590 A1 5, 1990 EP 1335,620 B1, 3, 2009 DE 4126902 A1 2/1992 EP 20698.95 A1 6, 2009 DE 4445O23 A1 6, 1996 EP 2 109. 295 A1 10, 2009 DE 102004O29203 A1 12/2005 EP 2109295 10/2009 DE 1984 1541 B4 12/2007 EP 172O375 B1 7, 2010 EP O057514 A1 8/1982 EP 2400373 A1 12/2011 EP 0.138061 A1 4, 1985 GB 2293667 A 4f1996 EP O218859 A2 4, 1987 GB 2310559 A 8, 1997 EP O262938 A1 4, 1988 GB 23428O2 A 4/2000 EP 02839.95 A2 9, 1988 GB 2346500 A 8, 2000 EP O293259 A2 11, 1988 GB 2352377 A 1, 2001 EP O299.572 A2 1, 1989 GB 2384399 A T 2003 EP O313975 A2 5, 1989 GB 2402855. A 12/2004 EP O314908 A2 5, 1989 GB 244.5436 A T 2008 EP O327408 A2 8, 1989 IT FI20010199 A1 4, 2003 EP O389.271 A2 9, 1990 JP 57-41731 U. 3, 1982 EP O411675 A2 2/1991 JP 59–57336 U. 4f1984 EP O441089 A2 8, 1991 JP 2-86.397. A 3, 1990 EP 0464712 A2 1/1992 JP 2-153415 A 6, 1990 EP O476.972 A2 3, 1992 JP 3-113578 A 5, 1991 EP O5583.12 A1 9, 1993 JP 4-236624 A 8, 1992 EP O559349 A1 9, 1993 EP O559349 B1 9, 1993 JP 5-79951 A 3, 1993 EP O570660 A1 11, 1993 JP 5-165459. A 7, 1993 EP 0575146 A2 12/1993 JP 5-293126. A 11, 1993 EP 0578604 A1 1/1994 JP 06 019965 1, 1994 EP O586996 A2 3, 1994 JP 6-69.954 A 3, 1994 EP 0609030 A1 8, 1994 JP 6-274586 A 9, 1994 EP O651543 A2 5, 1995 JP 6-332617. A 12, 1994 EP O679005 A1 10, 1995 JP 2007-1993.79 A 8, 1995 US 9,318.108 B2 Page 25

(56) References Cited JP 2010-287063 A 12/2010 JP 2013-511214. A 3, 2013 FOREIGN PATENT DOCUMENTS KR 10-1999-0073234. A 10, 1999 KR 11-2002-0013984. A 2, 2002 JP 7-320051 A 12/1995 KR 10-2002-0057.262 A T 2002 JP T-320079 A 12/1995 KR 10-2002-0069952 A 9, 2002 JP 8-63330 A 3, 1996 KR 10-2003-0016993 A 3 (2003 JP 8-185265 A T 1996 KR 10-2004-0044632 A 5, 2004 JP 8-223281 A 8, 1996 KR 10-2005-0083561 A 8, 2005 JP 8-227341. A 9, 1996 KR 10-2005-0090568 A 9, 2005 JP 9-18585 A 1, 1997 KR 10-2006-0011603. A 2, 2006 JP 9-55792 A 2, 1997 KR 10-2006-001.2730 A 2, 2006 JP 9-259063 A 10, 1997 KR 10-2006-0073574. A 6, 2006 JP 9-2654.57 A 10, 1997 KR 10-2007-0O24262. A 3 (2007 JP 1O-31497 A 2, 1998 KR 10-2007-0057496 6, 2007 JP 10-105324 A 4f1998 KR 10-2007-0071675 A 7/2007 JP 11-6743. A 1, 1999 KR 10-2007-010O837. A 10, 2007 JP 11-45241 A 2, 1999 KR 10-0776800 B1 11, 2007 JP 2000-0901 19 A 3, 2000 KR 10-2008-001.227 2, 2008 JP 2000-099225. A 4, 2000 KR 10-0810500 B1, 3, 2008 JP 2000-134407 A 5, 2000 KR 10-2008-0049647. A 6, 2008 JP 2000-163031 A 6, 2000 KR 10 2008 109322 A 12/2008 JP 2000-207167 A T 2000 KR 10-2009-0001716 A 1, 2009 JP 2000-224663. A 8, 2000 KR 10 2009 086805. A 8, 2009 JP 2000-339137 A 12/2000 KR 10-092O267 B1 10, 2009 JP 2001-56233 A 2, 2001 KR 10-2010-0032792 4/2010 JP 2001 125896 5, 2001 KR 10-2010-01 19519 A 11, 2010 JP 2001-148899. A 5, 2001 KR 10 2011 0113414. A 10, 2011 JP 2002-014954. A 1, 2002 KR 10-1193668 B1 12/2012 JP 2002 024.212 1, 2002 NL 1014847 C1 10, 2001 JP 2002-041624 A 2, 2002 RU 2273.106 C2 3, 2006 JP 2002-082893. A 3, 2002 RU 2349.970 C2 3, 2009 JP 2002-342033. A 11, 2002 RU 2353068 C2 4, 2009 JP 2002-344880 A 11, 2002 TW 2006.43744. A 12, 2006 JP 2002-542501. A 12/2002 TW 2008O1988 A 1, 2008 JP 2003-044.091 A 2, 2003 WO 93.2064.0 A1 10, 1993 JP 2003-84877. A 3, 2003 WO 94f16434 A1 7, 1994 JP 2003517158 (A) 5, 2003 WO 94,29788 A1 12, 1994 JP 2003-233568 A 8, 2003 WO WO95/02221 1, 1995 JP 2004-48804. A 2, 2004 WO 95/1695.0 A1 6, 1995 JP 2004-505525 A 2, 2004 WO 95/17746 A1 6, 1995 JP 2004-086356 A 3, 2004 WO 97.10586 A1 3/1997 JP 2004-152063. A 5, 2004 WO WO 97.26612 7/1997 JP 2005-07O645. A 3, 2005 WO 97.29614 A1 8/1997 JP 2005-86624 A 3, 2005 WO 97.38488 A1 10, 1997 JP 2005-5066O2 A 3, 2005 WO 97,49044 A1 12/1997 JP 2005-92441. A 4, 2005 WO 98,09270 A1 3/1998 JP 2005-181386. A 7/2005 WO 98.33111 A1 T 1998 JP 2005-189454 A 7/2005 WO WO98. 41956 9, 1998 JP 2005-221678 A 8, 2005 WO WO99,01834 1, 1999 JP 2005-283843 A 10, 2005 WO WO99/08238 2, 1999 JP 2005-31 1864. A 11, 2005 WO 99.16181 A1 4, 1999 JP 2006-023860. A 1, 2006 WO WO 99,56227 11, 1999 JP 2006-107438 A 4, 2006 WO 00, 19697 A1 4, 2000 JP 2006-146008 A 6, 2006 WO 00,22820 A1 4, 2000 JP 2006-195637 A T 2006 WO 00,29964 A1 5.2000 JP 2007-004633 A 1, 2007 WO OO,30070 A2 5, 2000 JP 2007-193794. A 8, 2007 WO 00,38041 A1 6, 2000 JP 2007-206317 A 8, 2007 WO 0044173 A1 7, 2000 JP 2008-26381 A 2, 2008 WO OO6376.6 A1 10, 2000 JP 2008-039928 A 2, 2008 WO WOOOf 6O435 10, 2000 JP 2008-090545 A 4/2008 WO WOOO60435 A3 10, 2000 JP 2008-97003. A 4/2008 WO OO68936 A1 11, 2000 JP 2008-134949. A 6, 2008 WO 01.06489 A1 1/2001 JP 2008-526101 A T 2008 WO O1/300.46 A2 4, 2001 JP 2008-217468 A 9, 2008 WO O1/30047 A2 4, 2001 JP 2008-233678 A 10, 2008 WO O1/33569 A1 5, 2001 JP 2008-236448 A 10, 2008 WO O1/35391 A1 5, 2001 JP 2008-271481. A 11, 2008 WO Of 46946 A1 6, 2001 JP 2009 036999 2, 2009 WO O1,65413 A1 9, 2001 JP 2009-047920. A 3, 2009 WO O1? 67753 A1 9, 2001 JP 2009-098490. A 5/2009 WO 02/2561.0 A1 3f2002 JP 2009-186989 A 8, 2009 WO 02A31814 A1 4, 2002 JP 2009-205367 A 9, 2009 WO O2.37469 A2 5, 2002 JP 2009-294913. A 12/2009 WO O2/O71259 A1 9, 2002 JP 2009-294.946. A 12/2009 WO O2O73603 9, 2002 JP 2010-078979 A 4, 2010 WO WO O2/O73603 A1 9, 2002 JP 2010-1572O7. A 7, 2010 WO O3,OO3152 A2 1/2003 JP 2010-535.377 A 1 1/2010 WO O3,OO3765 A1 1/2003 US 9,318.108 B2 Page 26

(56) References Cited Conference on Ontologies, DataBases, and Applications of Seman tics (ODBASE), Agia Napa, Cyprus, ttp://www.isi.edu/people/ FOREIGN PATENT DOCUMENTS ambite/publications/integration heterogeneous knowledge Sources calo query manager, 18 pages. WO 03/023786 A2 3, 2003 Belvin, R. et al., “Development of the HRL Route Navigation Dia WO 03/041364 A2 5, 2003 logue System.” 2001. In Proceedings of the First International Con WO 03/049494 A1 6, 2003 ference on Human Language Technology Research, Paper, Copyright WO O3,056789 A1 T 2003 WO O3,0672O2 A2 8, 2003 (C) 2001 HRL Laboratories, LLC, http://citeseerx.ist.psu.edu/ WO O3,084.196 A1 10, 2003 viewdoc/summary?doi=10.1.1.10.6538, 5 pages. WO 03/094489 A1 11 2003 Berry, P. M., et al. “PTIME: Personalized Assistance for Calendar WO 2004/008801 A1 1, 2004 ing.” ACM Transactions on Intelligent Systems and Technology, vol. WO 2004/O25938 A1 3, 2004 2, No. 4. Article 40, Publication date: Jul. 2011, 40: 1-22, 22 pages. WO 2004/047415 A1 6, 2004 Butcher, M.. “EVI arrives in town to go toe-to-toe with Siri.” Jan. 23. WO 2004/055637 A2 T 2004 2012, http://techcrunch.com/2012/01/23/evi-arrives-in-town-to-go WO 2004/057486 A1 T 2004 toe-to-toe-with-siri?, 2 pages. WO 2004/061850 A1 T 2004 Chen, Y., “Multimedia Siri Finds And Plays Whatever You Ask For.” WO 2004/084413 A2 9, 2004 Feb. 9, 2012, http://www.psfk.com/2012/02/multimedia-siri.html, 9 WO 2005/OO3920 A2 1, 2005 pageS. WO 2005/008505 A1 1, 2005 WO 2005/OO8899 A1 1, 2005 Cheyer, A. et al., “Spoken Language and Multimodal Applications WO 2005/O10725 A2 2, 2005 for Electronic Realties.” (C) Springer-Verlag London Ltd, Virtual WO 2005/027472 A2 3, 2005 Reality 1999, 3:1-15, 15 pages. WO 2005/027485 A1 3, 2005 Cutkosky, M. R. et al., “PACT: An Experiment in Integrating Con WO 2005/031737 A1 4/2005 current Engineering Systems.” Journal, Computer, vol. 26 Issue 1. WO 2005/034085 A1 4/2005 Jan. 1993, IEEE Computer Society Press Los Alamitos, CA, USA, WO 2005/041455 A1 5, 2005 http://dl.acm.org/citation.cfm?id=165320, 14 pages. WO 2005/059895 A1 6, 2005 Elio, R. et al., “On Abstract Task Models and Conversation Policies.” WO 2005/069171 A1 7/2005 http://webdocs.cs.ualberta.ca/~ree/publications/paperS2/ATS. WO 2005,101,176 A2 10/2005 AA99.pdf, 10 pages. WO 2006/02O305 A2 2, 2006 WO 2006/054724 A1 5, 2006 Ericsson, S. et al., “Software illustrating a unified approach to WO 2006/056822 A1 6, 2006 multimodality and multilinguality in the in-home domain.” Dec. 22. WO 2006/078246 A1 T 2006 2006, Talk and Look: Tools for Ambient Linguistic Knowledge, WO 2006,101649 A2 9, 2006 http://www.talk-project.eurice.eu/fileadmin?talk/publications pub WO 2006,133571 A1 12/2006 lic/deliverables public/D1 6.pdf, 127 pages. WO WO 2006/129967 A1 12/2006 Evi, “Meet Evil: the one mobile app that provides solutions for your WO 2007/002753 A2 1, 2007 everyday problems.” Feb. 8, 2012, http://www.evi.com/, 3 pages. WO 2007/083894 A1 7/2007 Feigenbaum, E., et al., “Computer-assisted Semantic Annotation of WO WO 2007080559 A2 7/2007 Scientific LifeWorks.” 2007. http://tomgruber.org/writing/stanford WO 2008/03097O A2 3, 2008 cs300.pdf, 22 pages. WO 2008/071231 A1 6, 2008 Gannes, L., “Alfred App Gives Personalized Restaurant Recommen WO 2008O85742 T 2008 dations,” althingsd.com, Jul. 18, 2011, http://allthingsd.com/ WO WO 2008/085.742 A2 T 2008 20110718/alfred-app-gives-personalized-restaurant-recommenda WO 20081098.35 9, 2008 WO WO 2008/109835 A2 9, 2008 tions, 3 pages. WO 2008. 140236 A1 11, 2008 Gautier, P. O., et al. “Generating Explanations of Device Behavior WO 2008,153639 A1 12/2008 Using Compositional Modeling and Causal Ordering.” 1993, http:// WO 2009/009240 A2 1, 2009 citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1. 142.8394. 9 WO 2009/O16631 A2 2, 2009 pageS. WO 2009/017280 A1 2, 2009 Gervasio, M.T. et al., Active Preference Learning for Personalized WO 2009/104126 A1 8, 2009 Calendar Scheduling Assistancae, Copyright (C) 2005, http://www.ai. WO 2009/156438 A1 12/2009 Sri.com/~gervasio/pubs/gervasio-iuiO5.pdf, 8 pages. WO 2010/075623 A1 T 2010 Glass, A., “Explaining Preference Learning.” 2006, http://cs229. WO 2011/057346 A1 5, 2011 Stanford.edu/proj2006/Glass-ExplainingPreferenceLearning.pdf, 5 WO 2011/060106 A1 5, 2011 WO WO 2011/088053 A2 T 2011 pageS. WO 2011 116309 A1 9, 2011 Gruber, T. R., et al., “An Ontology for Engineering Mathematics.” In WO 2011, 133543 A1 10, 2011 Jon Doyle, Piero Torasso, & Erik Sandewall, Eds. Fourth Interna WO 2011 150730 A1 12/2011 tional Conference on Principles of Knowledge Representation and WO 2011, 163350 A1 12/2011 Reasoning, Gustav Stresemann Institut, Bonn, Germany, Morgan WO 2012/154317 A1 11, 2012 Kaufmann, 1994, http://www-kslistanford.edu/knowledge-sharing? WO WO2012/1671.68 A2 12/2012 papers/engmath.html, 22 pages. WO 2013/048.880 A1 4/2013 Gruber, T. R. “A Translation Approach to Portable Ontology Speci fications.” Knowledge Systems Laboratory, Stanford University, OTHER PUBLICATIONS Sep. 1992, Technical Report KSL 92-71, Revised Apr. 1993, 27 EP Communication under Rule-161(2) and 162 EPC for Application pageS. No. 1170793922-2201, 4 pages. Gruber, T. R. “Automated Knowledge Acquisition for Strategic International Search Report and Written Opinion dated Nov. 29. Knowledge.” Knowledge Systems Laboratory, Machine Learning, 4. 2011, received in International Application No. PCT/US2011/ 293-336 (1989), 44 pages. 020861, which corresponds to U.S. Appl. No. 12987,982, 15 pages Gruber, T. R., "(Avoiding) the Travesty of the Commons.” Presenta (Gruber). tion at NPUC 2006, New Paradigms for User Computing, IBM Alfred App, 2011. http://www.alfredapp.com/, 5 pages. Almaden Research Center, Jul. 24, 2006. http://tomgruber.org/writ Ambite, JL., et al., “Design and Implementation of the CALO Query ing avoiding-travestry.htm, 52 pages. Manager.” Copyright (C) 2006, American Association for Artificial Gruber, T. R. “Big Think Small Screen: How semantic computing in Intelligence, (www.aaai.org), 8 pages. the cloud will revolutionize the consumer experience on the phone.” Ambite, JL., et al., “Integration of Heterogeneous Knowledge Keynote presentation at Web 3.0 conference, Jan. 27, 2010, http:// Sources in the CALO Query Manager.” 2005. The 4th International tomgruber.org/writing/web30an2010.htm, 41 pages. US 9,318.108 B2 Page 27

(56) References Cited from Proceedings of the Second International Conference on Artifi cial Intelligence in Design, Pittsburgh, USA, Jun. 22-25, 1992, 19 OTHER PUBLICATIONS pageS. Gruber, T. R. “Toward Principles for the Design of Ontologies Used Gruber, T. R. "Collaborating around Shared Content on the WWW.” for Knowledge Sharing.” In International Journal Human-Computer W3C Workshop on WWW and Collaboration, Cambridge, MA, Sep. Studies 43, p. 907-928, Substantial revision of paper presented at the 11, 1995, http://www.w3.org/Collaboration/Workshop/Proceedings/ International Workshop on Formal Ontology, Mar. 1993, Padova, P9.html, 1 page. Italy, available as Technical Report KSL93-04, Knowledge Systems Gruber, T. R. "Collective Knowledge Systems: Where the Social Laboratory, Stanford University, further revised Aug. 23, 1993, 23 Web meets the Semantic Web.' Web Semantics: Science, Services and Agents on the World WideWeb (2007), doi:10.1016/j.websem. pageS. 2007.11.011, keynote presentation given at the 5th International Guzzoni. D., et al., “Active, A Platform for Building Intelligent Semantic Web Conference, Nov. 7, 2006, 19 pages. Operating Rooms.” Surgetica 2007 Computer-Aided Medical Inter Gruber, T. R. “Where the Social Web meets the Semantic Web. ventions: tools and applications, pp. 191-198, Paris, 2007, Sauramps Presentation at the 5th International Semantic Web Conference, Nov. Medical, http://lsroepfl.ch/page-68384-en.html, 8 pages. 7, 2006, 38 pages. Guzzoni. D., et al., “Active, A Tool for Building Intelligent User Gruber, T. R., “Despite our Best Efforts, Ontologies are not the Interfaces.” ASC 2007, Palma de Mallorca, http://lsroepfl.ch/page Problem.” AAAI Spring Symposium, Mar. 2008, http://tomgruber. 34241.html, 6 pages. org/writing?aaai-SS08.htm, 40 pages. Hardawar, D., “Driving app Waze builds its own Siri for hands-free Gruber, T. R., “Enterprise Collaboration Management with voice control.” Feb. 9, 2012, http://venturebeat.com/2012/02/09/ Intraspect.” Intraspect Software, Inc., Instraspect Technical White driving-app-waZe-builds-its-own-siri-for-hands-free-voice-control/, Paper Jul. 2001, 24 pages. 4 pages. Gruber, T. R. “Every ontology is a treaty—a social agreement— Intraspect Software, “The Intraspect Knowledge Management Solu among people with some common motive in sharing.” Interview by tion: Technical Overview.” http://tomgruber.org/writing/intraspect Dr. Miltiadis D. Lytras, Official Quarterly Bulletin of AIS Special whitepaper-1998.pdf, 18 pages. Interest Group on Semantic Web and Information Systems, vol. 1, Julia, L., et al., Un éditeur interactif detableaux dessinés a mainlevée Issue 3, 2004, http://www.sigsemis.org 1, 5 pages. (An Interactive Editor for Hand-Sketched Tables), Traitement du Gruber, T. R. et al., “Generative Design Rationale: Beyond the Signal 1995, vol. 12, No. 6, 8 pages. No English Translation Avail Record and Replay Paradigm.” Knowledge Systems Laboratory, able. Stanford University, Dec. 1991, Technical Report KSL 92-59, Karp, P. D., “A Generic Knowledge-Base Access Protocol.” May 12, Updated Feb. 1993, 24 pages. 1994, http://lecture.cs.buu.ac.th/-f50353/Document/gfp.pdf, 66 Gruber, T. R. “Helping Organizations Collaborate, Communicate, pageS. and Learn.” Presentation to NASA Ames Research, Mountain View, Lemon, O., et al., “Multithreaded Context for Robust Conversational CA, Mar. 2003, http://tomgruber.org/writing/organizational-intelli Interfaces: Context-Sensitive Speech Recognition and Interpretation gence-talk.htm, 30 pages. of Corrective Fragments.” Sep. 2004, ACM Transactions on Com Gruber, T. R., “Intelligence at the Interface: Semantic Technology puter-Human Interaction, vol. 11, No. 3, 27 pages. and the Consumer Internet Experience.” Presentation at Semantic Leong, L., et al., “CASIS: A Context-Aware Speech Interface Sys Technologies conference (SemTech08), May 20, 2008, http:// tem.” IUI'05, Jan. 9-12, 2005, Proceedings of the 10th international tomgruber.org/writing.htm, 40 pages. conference on Intelligent user interfaces, San Diego, California, Gruber, T. R., Interactive Acquisition of Justifications: Learning USA, 8 pages. “Why” by Being Told “What” Knowledge Systems Laboratory, Lieberman, H., et al., “Out of context: Computer systems that adapt Stanford University, Oct. 1990, Technical Report KSL 91-17, to, and learn from, context.” 2000, IBM Systems Journal, vol. 39. Revised Feb. 1991, 24 pages. Nos. 3/4, 2000, 16 pages. Gruber, T. R. “It Is What It Does: The Pragmatics of Ontology for Lin, B., et al., “A Distributed Architecture for Cooperative Spoken Knowledge Sharing” (c) 2000, 2003, http://www.cidoc-crim.org/ Dialogue Agents with Coherent Dialogue State and History.” 1999, docs/symposium presentations/gruber cidoc-ontology-2003.pdf http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.42.272, 4 21 pages. pageS. Gruber, T. R., et al., “Machine-generated Explanations of Engineer McGuire, J., et al., “Shade: Technology for Knowledge-Based Col ing Models: A Compositional Modeling Approach.” (1993) In Proc. laborative Engineering.” 1993, Journal of Concurrent Engineering: International Joint Conference on Artificial Intelligence, http:// Applications and Research (CERA), 18 pages. citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.34.930, 7 Milward, D., et al., “D2.2: Dynamic Multimodal Interface pageS. Reconfiguration.” Talk and Look: Tools for Ambient Linguistic Gruber, T. R. "2021: Mass Collaboration and the Really New Knowledge, Aug. 8, 2006, http://www.ihmc.us/users/nblaylock? Economy.” TNTY Futures, the newsletter of The Next Twenty Years Pubs/Files/talk d2.2.pdf. 69 pages. series, vol. 1, Issue 6, Aug. 2001, http://www.tnty.com/newsletter? Mitra, P. et al., “A Graph-Oriented Model for Articulation of Ontol futures/archive/vol1-05business.html, 5 pages. ogy Interdependencies,” 2000, http://ilpubs. Stanford.edu:8090/442/ Gruber, T. R. et al..."Nike: A National Infrastructure for Knowledge 1/2000-20.pdf, 15 pages. Exchange.” Oct. 1994, http://www.eit.com/papers/nike?nike.html Moran, D. B., et al., “Multimodal User Interfaces in the Open Agent and nike.ps, 10 pages. Architecture.” Proc. of the 1997 International Conference on Intelli Gruber, T. R. “Ontologies, Web 2.0 and Beyond.” Apr. 24, 2007. gent User Interfaces (IUI97), 8 pages. Ontology Summit 2007. http://tomgruber.org/writing? ontolog-So Mozer, M., “An Intelligent Environment Must be Adaptive.” Mar? cial-web-keynote.pdf, 17 pages. Apr. 1999, IEEE Intelligent Systems, 3 pages. Gruber, T. R. “Ontology of Folksonomy: A Mash-up of Apples and Mühlhäuser, M., "Context Aware Voice User Interfaces for Workflow Oranges.” Originally published to the web in 2005, Int'l Journal on Support.” Darmstadt 2007. http://tuprints.ulb.tu-darmstadt.de/876/1/ Semantic Web & Information Systems, 3(2), 2007, 7 pages. PhD.pdf, 254 pages. Gruber, T. R. “Siri, a Virtual Personal Assistant Bringing Intelli Naone, E., “TR10: Intelligent Software Assistant.” Mar.-Apr. 2009, gence to the Interface.” Jun. 16, 2009, Keynote presentation at Technology Review, http://www.technologyreview.com/printer , Semantic Technologies conference, Jun. 2009. http://tomgruber.org/ friendly article.aspx?id=22117, 2 pages. Writing semtech09.htm, 22 pages. Neches, R., “Enabling Technology for Knowledge Sharing.” Fall Gruber, T. R. “TagOntology.” Presentation to Tag Camp, www. 1991, Al Magazine, pp. 37-56, (21 pages). tagcamp.org, Oct. 29, 2005, 20 pages. Nöth, E., et al., “Verbmobil: The Use of Prosody in the Linguistic Gruber, T.R., et al., “Toward a KnowledgeMedium for Collaborative Components of a Speech Understanding System.” IEEE Transactions Product Development.” In Artificial Intelligence in Design 1992, on Speech and Audio Processing, vol. 8, No. 5, Sep. 2000, 14 pages. US 9,318.108 B2 Page 28

(56) References Cited Alshawi H., et al., “Logical Forms in the Core Language Engine.” 1989, Proceedings of the 27th Annual Meeting of the Association for OTHER PUBLICATIONS Computational Linguistics, 8 pages. Alshawi, H., et al., “Overview of the Core Language Engine.” Sep. Rice, J., et al., “Monthly Program: Nov. 14, 1995.” The San Francisco 1988, Proceedings of Future Generation Computing Systems, Tokyo, Bay Area Chapter of ACM SIGCHI, http://www.baychi.org/calen 13 pages. dar/19951114?, 2 pages. Alshawi, H., “Translation and Monotonic Interpretation/Genera Rice, J., et al., “Using the Web Instead of a Window System.” Knowl tion.” Jul. 1992, SRI International, Cambridge Computer Science edge Systems Laboratory, Stanford University, http://tomgruber.org/ Research Centre, Cambridge, 18 pages, http://www.cam. Sri.com.tr/ writing?ksl-95-69.pdf, 14 pages. circO24/paperps.Z 1992. Rivlin, Z. et al., “Maestro: Conductor of Multimedia Analysis Tech Appelt, D., et al., “Fastus: A Finite-state Processor for Information nologies.” 1999 SRI International, Communications of the Associa Extraction from Real-world Text,” 1993, Proceedings of IJCAI, 8 tion for Computing Machinery (CACM), 7 pages. pageS. Sheth, A., et al., “Relationships at the Heart of Semantic Web: Mod Appelt, D., et al., “SRI: Description of the JV-FASTUS System Used eling, Discovering, and Exploiting Complex Semantic Relation for MUC-5, 1993, SRI International, Artificial Intelligence Center, ships.” Oct. 13, 2002, Enhancing the Power of the Internet: Studies in 19 pages. FuZZiness and Soft Computing, Springer Verlag, 38 pages. Appelt, D., et al., SRI International Fastus System MUC-6 Test Simonite, T., "One EasyWay to Make Siri Smarter.” Oct. 18, 2011, Results and Analysis, 1995, SRI International, Menlo Park, Califor Technology Review, http://www.technologyreview.com/printer nia, 12 pages. friendly article.aspx?id=38915, 2 pages. Archbold, A., et al., "A Team User's Guide.” Dec. 21, 1981, SRI Stent, A., et al., “The Command Talk Spoken Dialogue System.” International, 70 pages. 1999, http://acl.ldc.upenn.edu/P/P99/P99-1024.pdf, 8 pages. Bear, J., et al., “A System for Labeling Self-Repairs in Speech.” Feb. Tofel, K., et al., “SpeakTolt: A personal assistant for older iPhones, 22, 1993, SRI International, 9 pages. iPads.” Feb. 9, 2012, http://gigaom.com/apple/speaktoit-siri-for Bear, J., et al., “Detection and Correction of Repairs in Human older-iphones-ipads, 7 pages. Computer Dialog.” May 5, 1992, SRI International, 11 pages. Tucker, J., “Too lazy to grabyour TV remote? Use Siri instead.” Nov. Bear, J., et al., “Integrating Multiple Knowledge Sources for Detec 30, 2011, http://www.engadget.com/2011/11/30/too-lazy-to-grab tion and Correction of Repairs in Human-Computer Dialog,” 1992, your-tv-remote-use-siri-instead?, 8 pages. Proceedings of the 30th annual meeting on Association for Compu Tur, G., et al., “The CALO Meeting Speech Recognition and Under tational Linguistics (ACL), 8 pages. standing System.” 2008, Proc. IEEE Spoken Language Technology Bear, J., et al., “Using Information Extraction to Improve Document Workshop, 4 pages. Retrieval, 1998, SRI International, Menlo Park, California, 11 pageS. Tur, G., et al., “The-CALO-Meeting-Assistant System.” IEEE Trans Berry, P. et al., "Task Management under Change and Uncertainty actions on Audio, Speech, and Language Processing, vol. 18, No. 6, Constraint Solving Experience with the CALO Project” 2005, Pro Aug. 2010, 11 pages. ceedings of CP05 Workshop on Constraint Solving under Change, 5 YouTube, “Knowledge Navigator.” 5:34 minute video uploaded to pageS. YouTube by Knownav on Apr. 29, 2008, http://www.youtube.com/ Bobrow, R. et al., “Knowledge Representation for Syntactic/Seman watch?v=QRH8eimU 20on Aug. 3, 2006, 1 page. tic Processing.” From: AAA-80 Proceedings. Copyright (C) 1980, YouTube, “Send Text, Listen to and Send E-Mail “By Voice’ www. AAAI, 8 pages. voiceassist.com.” 2:11 minute video uploaded to YouTube by Bouchou, B., et al., “Using Transducers in Natural Language Data VoiceAssist on Jul. 30, 2009, http://www.youtube.com/ base Query.” Jun. 17-19, 1999, Proceedings of 4th International watch?v=0teU61nHHA4, 1 page. Conference on Applications of Natural Language to Information YouTube, “TextinDrive App Demo–Listen and Reply to your Mes Systems, Austria, 17 pages. sages by Voice while Driving,” 1:57 minute video uploaded to Bratt, H., et al., “The SRI Telephone-based ATIS System.” 1995, YouTube by TextinDrive on Apr. 27, 2010, http://www.youtube.com/ Proceedings of ARPA Workshop on Spoken LanguageTechnology,3 watch?v=WaGfzoHSAMw, 1 page. pageS. YouTube. “Voice on the Go (BlackBerry).” 2:51 minute video Bulyko, I. et al., “Error-Correction Detection and Response Genera uploaded to YouTube by Voice0nTheGo on Jul. 27, 2009, http:// tion in a Spoken Dialogue System.” (C) 2004 Elsevier B.V., specom. www.youtube.com/watch?v=p.JqpWgQS98w, 1 page. 2004.09.009, 18 pages. International Search Report and Written Opinion dated Nov. 29. Burke, R., et al., “Question Answering from Frequently Asked Ques 2011, received in International Application No. PCT/US2011/20861, tion Files.” 1997, AI Magazine, vol. 18, No. 2, 10 pages. which corresponds to U.S. Appl. No. 12987,982, 15 pages (Thomas Burns, A., et al., “Development of a Web-Based Intelligent Agent for Robert Gruber). the Fashion Selection and Purchasing Process via Electronic Com Guzzoni, Didier, et al., “Modeling Human-Agent Interaction with merce.” Dec. 31, 1998, Proceedings of the Americas Conference on Active Ontologies'. American Association for Artificial Intelligence, Information system (AMCIS), 4 pages. 2007, 8 pages. Carter, D., "Lexical Acquisition in the Core Language Engine.” 1989, Keleher, Erin, et al., "Vlingo Launches Voice Enablement Applica Proceedings of the Fourth Conference on the European Chapter of the tion of Apple App Store'. Cambridge, Mass., Dec. 3, 2008, www. Association for Computational Linguistics, 8 pages. vlingo.com. 2 pages. Carter, D., et al., “The Speech-Language Interface in the Spoken Agnäs, MS., et al., “Spoken Language Translator: First-Year Report.” Language Translator.” Nov. 23, 1994. SRI International, 9 pages. Jan. 1994, SICS (ISSN 0283-3638), SRI and Telia Research AB, 161 Chai, J., et al., “Comparative Evaluation of a Natural Language pageS. Dialog Based System and a Menu Driven System for Information Allen, J., “Natural Language Understanding.” 2nd Edition, Copy Access: a Case Study.” Apr. 2000, Proceedings of the International right (C) 1995 by the Benjamin/Cummings Publishing Company, Inc., Conference on Multimedia Information Retrieval (RIAO), Paris, 11 671 pages. pageS. Alshawi, H., et al., “CLARE: A Contextual Reasoning and Coopera Cheyer, A., et al., “Multimodal Maps: An Agent-based Approach.” tive Response Framework for the Core Language Engine.” Dec. International Conference on Cooperative Multimodal Communica 1992, SRI International, Cambridge Computer Science Research tion, 1995, 15 pages. Centre, Cambridge, 273 pages. Cheyer, A., et al., “The Open Agent Architecture. Autonomous Alshawi, H., et al., “Declarative Derivation of Database Queries from Agents and Multi-Agent systems, vol. 4, Mar. 1, 2001, 6 pages. Meaning Representations.” Oct. 1991, Proceedings of the Bankai Cheyer, A., et al., “The Open Agent Architecture: Building commu Workshop on Intelligent Information Access, 12 pages. nities of distributed software agents' Feb. 21, 1998, Artificial Intel US 9,318.108 B2 Page 29

(56) References Cited GroSZ, B. et al., “Dialogic: A Core Natural-Language Processing System.” Nov. 9, 1982, SRI International, 17 pages. OTHER PUBLICATIONS Grosz, B. et al., “Research on Natural-Language Processing at SRI.” Nov. 1981, SRI International, 21 pages. ligence Center SRI International, Power Point presentation, down Grosz, B., et al., “Team: An Experiment in the Design of Transport loaded from http://www.ai. Sri.com/~oaa?, 25 pages. able Natural-Language Interfaces.” Artificial Intelligence, vol. 32. Codd, E. F., “Databases: Improving Usability and Responsiveness— 1987, 71 pages. 'How About Recently.” Copyright (C) 1978, by Academic Press, Inc., GroSZ, B., “Team: A Transportable Natural-Language Interface Sys 28 pages. tem.” 1983, Proceedings of the First Conference on Applied Natural Cohen, P.R., et al., “An Open Agent Architecture.” 1994, 8 pages. Language Processing, 7 pages. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.30,480. Guida, G., et al., “NLI: A Robust Interface for Natural Language Coles, L. S., et al., “Chemistry Question-Answering.” Jun. 1969, SRI Person-Machine Communication.” Int. J. Man-Machine Studies, vol. International, 15 pages. 17, 1982, 17 pages. Guzzoni. D., et al., “Active. A platform for Building Intelligent Soft Coles, L. S., “Techniques for Information Retrieval Using an Infer ware.” Computational Intelligence 2006, 5 pages. http://www. ential Question-Answering System with Natural-Language Input.” informatik.uni-trier.de/-ley pers/hd/g/Guzzoni:Didier. Nov. 1972, SRI International, 198 Pages. Guzzoni. D., “Active: A unified platform for building intelligent Coles, L. S., “The Application of Theorem Proving to Information assistant applications.” Oct. 25, 2007, 262 pages. Retrieval.” Jan. 1971, SRI International, 21 pages. Guzzoni. D., et al., “A Unified Platform for Building Intelligent Web Constantinides, P. et al., “A Schema Based Approach to Dialog Interaction Assistants.” Proceedings of the 2006 IEEE/WIC/ACM Control.” 1998, Proceedings of the International Conference on Spo International Conference on Web Intelligence and Intelligent Agent ken Language Processing, 4 pages. Technology, Computer Society, 4 pages. Cox, R. V., et al., “Speech and Language Processing for Next-Mil Guzzoni, D., et al., “Many Robots Make Short Work.” 1996 AAAI lennium Communications Services.” Proceedings of the IEEE, vol. Robot Contest, SRI International, 9 pages. 88, No. 8, Aug. 2000, 24 pages. Haas, N., et al., “An Approach to Acquiring and Applying Knowl Craig, J., et al., “Deacon: Direct English Access and Control.” Nov. edge.” Nov. 1980, SRI International, 22 pages. 7-10, 1966 AFIPS Conference Proceedings, vol. 19, San Francisco, Hadidi, R., et al., “Students' Acceptance of Web-Based Course Offer 18 pages. ings: An Empirical Assessment.” 1998 Proceedings of the Americas Dar, S., et al., “DTL's DataSpot: Database Exploration Using Plain Conference on Information Systems (AMCIS), 4 pages. Language.” 1998 Proceedings of the 24th VLCB Conference, New Hawkins, J., et al., “Hierarchical Temporal Memory: Concepts, York, 5 pages. Theory, and Terminology.” Mar. 27, 2007. Numenta, Inc., 20 pages. Davis, Z. et al., “A Personal Handheld Multi-Modal Shopping Assis He, Q., et al., “Personal Security Agent: KQML-Based PKI.” The tant.” 2006 IEEE, 9 pages. Robotics Institute, Carnegie-Mellon University, paper, Oct. 1, 1997. Decker, K., et al., “Designing Behaviors for Information Agents.” 14 pages. The Robotics Institute, Carnegie-Mellon University, paper, Jul. 6, Hendrix, G. et al., “Developing a Natural Language Interface to 1996, 15 pages. Complex Data.” ACM Transactions on Database Systems, vol. 3, No. Decker, K., et al., “Matchmaking and Brokering.” The Robotics 2, Jun. 1978, 43 pages. Institute, Carnegie-Mellon University, paper, May 16, 1996, 19 Hendrix, G., “Human Engineering for Applied Natural Language pageS. Processing.” Feb. 1977. SRI International, 27 pages. Dowding, J., et al., "Gemini: A Natural Language System for Spo Hendrix, G., “Klaus: A System for Managing Information and Com ken-Language Understanding.” 1993, Proceedings of the Thirty-First putational Resources.” Oct. 1980, SRI International, 34 pages. Annual Meeting of the Association for Computational Linguistics, 8 Hendrix, G., “Lifer: A Natural Language Interface Facility.” Dec. pageS. 1976, SRI Stanford Research Institute, Artificial Intelligence Center, Dowding, J., et al., “Interleaving Syntax and Semantics in an Efficient 9 pages. Bottom-Up Parser.” 1994, Proceedings of the 32nd Annual Meeting Hendrix, G., “Natural-Language Interface.” Apr.-Jun. 1982, Ameri of the Association for Computational Linguistics, 7 pages. can Journal of Computational Linguistics, vol. 8, No. 2, 7 pages. Epstein, M., et al., “Natural Language Access to a Melanoma Data Hendrix, G., “The Lifer Manual: A Guide to Building Practical Base.” Sep. 1978, SRI International, 7 pages. Natural Language Interfaces.” Feb. 1977. SRI International, 76 Exhibit 1. “Natural Language Interface Using Constrained Interme pageS. diate Dictionary of Results.” Classes/Subclasses Manually Reviewed Hendrix, G., et al., “Transportable Natural-Language Interfaces to for the Search of U.S. Pat. No. 7, 177,798, Mar. 22, 2013, 1 page. Databases.” Apr. 30, 1981, SRI International, 18 pages. Exhibit 1. “Natural Language Interface Using Constrained Interme Hirschman, L., et al., “Multi-Site Data Collection and Evaluation in diate Dictionary of Results.” List of Publications Manually reviewed Spoken Language Understanding.” 1993, Proceedings of the work for the Search of U.S. Pat. No. 7, 177,798, Mar. 22, 2013, 1 page. shop on Human Language Technology, 6 pages. Ferguson, G., et al., “Trips: An Integrated Intelligent Problem-Solv Hobbs, J., et al., “Fastus: A System for Extracting Information from ing Assistant.” 1998, Proceedings of the Fifteenth National Confer Natural-LanguageText.” Nov. 19, 1992, SRI International, Artificial ence on Artificial Intelligence (AAAI-98) and Tenth Conference on Intelligence Center, 26 pages. Innovative Applications of Artificial Intelligence (IAAI-98), 7 pages. Hobbs, J., et al., “Fastus: Extracting Information from Natural-Lan Fikes, R., et al., “A Network-based knowledge Representation and its guageTexts.” 1992, SRI International, Artificial Intelligence Center, Natural Deduction System.” Jul. 1977. SRI International, 43 pages. 22 pages. Gambäck, B., et al., “The Swedish Core Language Engine.” 1992 Hobbs, J., “Sublanguage and Knowledge.” Jun. 1984. SRI Interna NOTEX Conference, 17 pages. tional, Artificial Intelligence Center, 30 pages. Glass, J., et al., “Multilingual Language Generation Across Multiple Hodjat, B., et al., “Iterative Statistical Language Model Generation Domains.” Sep. 18-22, 1994. International Conference on Spoken for Use with an Agent-Oriented Natural Language Interface.” Vol. 4 Language Processing, Japan, 5 pages. of the Proceedings of HCI International 2003, 7 pages. Green, C. “The Application of Theorem Proving to Question-An Huang. X., et al., “The SPHINX-II Speech Recognition System: An Swering Systems.” Jun. 1969, SRI Stanford Research Institute, Arti Overview.” Jan. 15, 1992, Computer, Speech and Language, 14 ficial Intelligence Group, 169 pages. pageS. Gregg, D. G., “DSS Access on the WWW: An Intelligent Agent Issar, S., et al., “CMU's Robust Spoken Language Understanding Prototype.” 1998 Proceedings of the Americas Conference on Infor System.” 1993, Proceedings of Eurospeech, 4 pages. mation Systems—Association for Information Systems, 3 pages. Issar, S., “Estimation of Language Models for New Spoken Lan Grishman, R., "Computational Linguistics: An Introduction.” (C) guage Applications.” Oct. 3-6, 1996, Proceedings of4th International Cambridge University Press 1986, 172 pages. Conference on Spoken language Processing, Philadelphia, 4 pages. US 9,318.108 B2 Page 30

(56) References Cited Minker, W., et al., “Hidden Understanding Models for Machine Translation.” 1999, Proceedings of ETRW on Interactive Dialogue in OTHER PUBLICATIONS Multi-Modal Systems, 4 pages. Modi. P. J., et al., "CMRadar: A Personal Assistant Agent for Calen Janas, J., “The Semantics-Based Natural Language Interface to Rela dar Management.” (C) 2004, American Association for Artificial Intel tional Databases.” (C) Springer-Verlag Berlin Heidelberg 1986, Ger ligence, Intelligent Systems Demonstrations, 2 pages. many, 48 pages. Moore, R., et al., "Combining Linguistic and Statistical Knowledge Johnson, J., “A Data Management Strategy for TransportableNatural Sources in Natural-Language Processing for ATIS.” 1995, SRI Inter Language Interfaces.” Jun. 1989, doctoral thesis Submitted to the national, Artificial Intelligence Center, 4 pages. Department of Computer Science, University of British Columbia, Moore, R. “Handling Complex Queries in a Distributed Data Base.” Canada, 285 pages. Oct. 8, 1979, SRI International, Artificial Intelligence Center, 38 Julia, L., et al., "http://www.speech. Sri.com/demos?atis.html.” 1997. pageS. Proceedings of AAAI, Spring Symposium, 5 pages. Moore, R., “Practical Natural-Language Processing by Computer.” Kahn, M., et al., “CoABS Grid Scalability Experiments.” 2003, Oct. 1981, SRI International, Artificial Intelligence Center, 34 pages. Autonomous Agents and Multi-Agent Systems, vol. 7, 8 pages. Moore, R., et al., “SRI's Experience with the ATIS Evaluation.” Jun. Kamel, M., et al., “A Graph Based Knowledge Retrieval System.” (C) 24-27, 1990, Proceedings of a workshop held at Hidden Valley, 1990 IEEE, 7 pages. Pennsylvania, 4 pages. Katz, B., "Annotating the World Wide Web Using Natural Lan Moore, et al., “The Information Warefare Advisor: An Architecture guage.” 1997. Proceedings of the 5th RIAO Conference on Computer for Interacting with Intelligent Agents Across the Web.” Dec. 31. Assisted Information Searching on the Internet, 7 pages. 1998 Proceedings of Americas Conference on Information Systems Katz, B., "A Three-Step Procedure for Language Generation.” Dec. (AMCIS), 4 pages. 1980, Massachusetts Institute of Technology, Artificial Intelligence Moore, R., “The Role of Logic in Knowledge Representation and Laboratory, 42 pages. Commonsense Reasoning.” Jun. 1982, SRI International, Artificial Kats, B., et al., “Exploiting Lexical Regularities in Designing Natural Intelligence Center, 19 pages. Language Systems.” 1988, Proceedings of the 12th International Moore, R. “Using Natural-Language Knowledge Sources in Speech Conference on Computational Linguistics, Coling'88, Budapest, Recognition.” Jan. 1999, SRI International, Artificial Intelligence Hungary, 22 pages. Center, 24 pages. Katz, B., et al., “Rextor: A System for Generating Relations from Moran, D., et al., “Intelligent Agent-based User Interfaces.” Oct. Natural Language.” In Proceedings of the ACL Oct. 2000 Workshop 12-13, 1995, Proceedings of International Workshop on Human on Natural Language Processing and Information Retrieval (NLP Interface Technology, University of Aizu, Japan, 4 pages. http:// &IR), 11 pages. www.dougmoran.com/dmoran/PAPERS/oaa-iwhit1995.pdf. Katz, B. “Using English for Indexing and Retrieving.” 1988 Pro Moran, D., "Quantifier Scoping in the SRI Core Language Engine.” ceedings of the 1st RIAO Conference on User-Oriented Content 1988, Proceedings of the 26th annual meeting on Association for Based Text and Image (RIAO'88), 19 pages. Computational Linguistics, 8 pages. Konolige, K., “A Framework for a Portable Natural-Language Inter Motro, A., “Flex: A Tolerant and Cooperative User Interface to Data face to Large DataBases.” Oct. 12, 1979, SRI International, Artificial bases.” IEEE Transactions on Knowledge and Data Engineering, vol. Intelligence Center, 54 pages. 2, No. 2, Jun. 1990, 16 pages. Laird, J., et al., “SOAR: An Architecture for General Intelligence.” Murveit, H., et al., “Speech Recognition in SRI's Resource Manage 1987. Artificial Intelligence vol. 33, 64 pages. ment and ATIS Systems.” 1991, Proceedings of the workshop on Langly, P. et al., “A Design for the Icarus Architechture.” SIGART Speech and Natural Language (HTL'91), 7 pages. Bulletin, vol. 2, No. 4, 6 pages. OAA, "The Open Agent Architecture 1.0 Distribution Source Code.” Larks, “Intelligent Software Agents: Larks.” 2006, downloaded on Copyright 1999, SRI International, 2 pages. Mar. 15, 2013 from http://www.cs.cmu.edu/larks.html, 2 pages. Odubiyi, J., et al., "SAIRE a scalable agent-based information Martin, D., et al., “Building Distributed Software Systems with the retrieval engine.” 1997 Proceedings of the First International Con Open Agent Architecture.” Mar. 23-25, 1998, Proceedings of the ference on Autonomous Agents, 12 pages. Third International Conference on the Practical Application of Intel Owei, V., et al., “Natural Language Query Filtration in the Concep ligent Agents and Multi-Agent Technology, 23 pages. tual Query Language.” (C) 1997 IEEE, 11 pages. Martin, D., et al., “Development Tools for the Open Agent Architec Pannu, A., et al., “A Learning Personal Agent for Text Filtering and ture.” Apr. 1996, Proceedings of the International Conference on the Notification.” 1996, The Robotics Institute School of Computer Sci Practical Application of Intelligent Agents and Multi-Agent Technol ence, Carnegie-Mellon University, 12 pages. ogy, 17 pages. Pereira, “Logic for Natural Language Analysis,” Jan. 1983, SRI Martin, D., et al., “Information Brokering in an Agent Architecture.” International, Artificial Intelligence Center, 194 pages. Apr. 1997. Proceedings of the second International Conference on Perrault, C.R., et al., “Natural-Language Interfaces.” Aug. 22, 1986, the Practical Application of Intelligent Agents and Multi-Agent Tech SRI International, 48 pages. nology, 20 pages. Pulman, S.G., et al., "Clare: A Combined Language and Reasoning Martin, D., et al., “PAAM '98 Tutorial: Building and Using Practical Engine.” 1993, Proceedings of JFIT Conference, 8 pages. URL: Agent Applications.” 1998, SRI International, 78 pages. http://www.cam. Sri.com.tr/crcC42/paperps.Z. Martin, P. et al., “Transportability and Generality in a Natural-Lan Ravishankar, “Efficient Algorithms for Speech Recognition.” May guage Interface System.” Aug. 8-12, 1983, Proceedings of the Eight 15, 1996, Doctoral Thesis submitted to School of Computer Science, International Joint Conference on Artificial Intelligence, West Ger Computer Science Division, Carnegie Mellon University, Pittsburg, many, 21 pages. 146 pages. Matiasek, J., et al., “Tamic-P: A System for NL Access to Social Rayner, M., 'Abductive Equivalential Translation and its application Insurance Database.” Jun. 17-19, 1999, Proceeding of the 4th Inter to Natural Language Database Interfacing.” Sep. 1993 Dissertation national Conference on Applications of Natural Language to Infor paper, SRI International, 163 pages. mation Systems, Austria, 7 pages. Rayner, M., et al., "Adapting the Core Language Engine to French Michos, S.E., et al., “Towards an adaptive natural language interface and Spanish.” May 10, 1996, Cornell University Library, 9 pages. to command languages.” Natural Language Engineering 2 (3), (C) http://arxiv.org/abs/cmp-lg/9605015. 1994 Cambridge University Press, 19 pages. Rayner, M., et al., “Deriving Database Queries from Logical Forms Milstead, J., et al., “Metadata: Cataloging by Any Other Name...” by Abductive Definition Expansion.” 1992, Proceedings of the Third Jan. 1999. Online, Copyright (C) 1999 Information Today, Inc., 18 Conference on Applied Natural Language Processing, ANLC'92, 8 pageS. pageS. US 9,318.108 B2 Page 31

(56) References Cited Wahlster, W., et al., “Smartkom: multimodal communication with a life-like character.” 2001 Eurospeech—Scandinavia, 7th European OTHER PUBLICATIONS Conference on Speech Communication and Technology, 5 pages. Waldinger, R., et al., “Deductive Question Answering from Multiple Rayner, M., "Linguistic Domain Theories: Natural-Language Data Resources.” 2003, New Directions in Question Answering, published base Interfacing from First Principles.” 1993, SRI International, by AAAI, Menlo Park, 22 pages. Cambridge, 11 pages. Walker, D., et al., “Natural Language Access to Medical Text.” Mar. Rayner, M., et al., “Spoken Language Translation With Mid-90's 1981, SRI International, Artificial Intelligence Center, 23 pages. Technology: A Case Study.” 1993, Eurospeech, ISCA, 4 pages. http:// Waltz, D., “An English Language Question Answering System for a dblp.uni-trier.de/db/confinterspeech/eurospeech 1993. Large Relational Database.” (C) 1978 ACM, vol. 21, No. 7, 14 pages. html#RaynerBCCDGKKLPPS93. Ward, W., et al., “A Class Based Language Model for Speech Rec Rudnicky, A.I., et al., “Creating Natural Dialogs in the Carnegie ognition.” (C) 1996 IEEE, 3 pages. Mellon Communicator System.”. Ward, W., et al., “Recent Improvements in the CMU Spoken Lan Russell, S., et al., “Artificial Intelligence, A Modern Approach.” (C) guage Understanding System.” 1994, ARPA Human Language Tech 1995 Prentice Hall, Inc., 121 pages. nology Workshop, 4 pages. Sacerdoti, E., et al., “A Ladder User's Guide (Revised).” Mar. 1980, Ward, W. “The CMUAirTravel Information Service: Understanding SRI International, Artificial Intelligence Center, 39 pages. Spontaneous Speech. 3 pages. Sagalowicz, D., “A D-Ladder User's Guide.” Sep. 1980, SRI Inter Warren, D.H.D., et al., “An Efficient Easily Adaptable System for national, 42 pages. Interpreting Natural Language Queries.” Jul.-Dec. 1982, American Sameshima, Y, et al., “Authorization with security attributes and Journal of Computational Linguistics, vol. 8, No. 3-4, 11 pages. privilege delegation Access control beyond the ACL. Computer Weizenbaum, J., “ELIZA-A Computer Program for the Study of Communications, vol. 20, 1997.9 pages. Natural Language Communication Between Man and Machine.” San-Segundo, R., et al., “Confidence Measures for Dialogue Man Communications of the ACM. vol. 9, No. 1, Jan. 1966, 10 pages. agement in the CUCommunicator System.” Jun. 5-9, 2000, Proceed Winiwarter, W., "Adaptive Natural Language Interfaces to FAQ ings of Acoustics, Speech, and Signal Processing (ICASSPOO), 4 Knowledge Bases.” Jun. 17-19, 1999, Proceedings of 4th Interna pageS. tional Conference on Applications of Natural Language to Informa Sato, H., “A Data Model, Knowledge Base, and Natural Language tion Systems, Austria, 22 pages. Processing for Sharing a Large Statistical Database.” 1989, Statistical Wu, X. et al., “KDA: A Knowledge-based Database Assistant.” Data and Scientific Database Management, Lecture Notes in Computer Engineering, Feb. 6-10, 1989, Proceeding of the Fifth International Science, vol. 339, 20 pages. Conference on Engineering (IEEE Cat. No. 89CH2695-5), 8 pages. Schnelle, D., "Context Aware Voice User Interfaces for Workflow Yang, J., et al., “Smart Sight: A Tourist Assistant System.” 1999 Support.” Aug. 27, 2007. Dissertation paper, 254 pages. Proceedings of Third International Symposium on Wearable Com Sharoff, S., et al., “Register-domain Separation as a Methodology for puters, 6 pages. Development of Natural Language Interfaces to Databases.” 1999, Zeng, D., et al., “Cooperative Intelligent Software Agents.” The Proceedings of Human-Computer Interaction (INTERACT'99), 7 Robotics Institute, Carnegie-Mellon University, Mar. 1995, 13 pages. pageS. Zhao, L., “Intelligent Agents for Flexible Workflow Systems.” Oct. Shimazu, H., et al., "CAPIT: Natural Language Interface Design Tool 31, 1998 Proceedings of the Americas Conference on Information with Keyword Analyzer and Case-Based Parser.” NEC Research & Systems (AMCIS), 4 pages. Development, vol. 33, No. 4, Oct. 1992, 11 pages. Zue, V., et al., “From Interface to Content: Translingual Access and Shinkle, L., “Team User's Guide.” Nov. 1984, SRI International, Delivery of On-Line Information.” 1997, Eurospeech, 4 pages. Artificial Intelligence Center, 78 pages. Zue, V., et al., “Jupiter: A Telephone-Based Conversational Interface Shklar, L., et al., “Info Harness: Use of Automatically Generated for Weather Information.” Jan. 2000, IEEE Transactions on Speech Metadata for Search and Retrieval of Heterogeneous Information.” and Audio Processing, 13 pages. 1995 Proceedings of CAiSE’95, Finland. Zue, V., et al., “Pegasus: A Spoken Dialogue Interface for On-Line Singh, N., “Unifying Heterogeneous Information Models.” 1998 Air Travel Planning.” 1994 Elsevier, Speech Communication 15 Communications of the ACM, 13 pages. (1994), 10 pages. SRI2009, “SRI Speech: Products: Software Development Kits: Zue, V., et al., “The Voyager Speech Understanding System: Prelimi EduSpeak.” 2009, 2 pages, available at http://web.archive.org/web/ nary Development and Evaluation.” 1990, Proceedings of IEEE 1990 20090828084033/http://www.speechatsri.com/products/eduspeak. International Conference on Acoustics, Speech, and Signal Process shtml. ing, 4 pages. Starr, B., et al., “Knowledge-Intensive Query Processing.” May 31, Australian Office Action dated Jul. 2, 2013 for Application No. 1998, Proceedings of the 5th KRDB Workshop, Seattle, 6 pages. 2011205426, 9 pages. Stern, R., et al. “Multiple Approaches to Robust Speech Recogni Certificate of Examination dated Apr. 29, 2013 for Australian Patent tion.” 1992, Proceedings of Speech and Natural LanguageWorkshop, No. 2012 101.191, 4 pages. 6 pages. Certificate of Examination dated May 21, 2013 for Australian Patent Stickel, “A Nonclausal Connection-Graph Resolution Theorem No. 2012101471, 5 pages. Proving Program.” 1982, Proceedings of AAAI'82, 5 pages. Certificate of Examination dated May 10, 2013 for Australian Patent Sugumaran, V., “A Distributed Intelligent Agent-Based Spatial Deci No. 2012101466, 4 pages. sion Support System.” Dec. 31, 1998, Proceedings of the Americas Certificate of Examination dated May 9, 2013 for Australian Patent Conference on Information systems (AMCIS), 4 pages. No. 2012101473, 4 pages. Sycara, K., et al., "Coordination of Multiple Intelligent Software Certificate of Examination dated May 6, 2013 for Australian Patent Agents.” International Journal of Cooperative Information Systems No. 2012101470, 5 pages. (IJCIS), vol. 5, Nos. 2 & 3, Jun. & Sep. 1996, 33 pages. Certificate of Examination dated May 2, 2013 for Australian Patent Sycara, K., et al., “Distributed Intelligent Agents.” IEEE Expert, vol. No. 2012101468, 5 pages. 11, No. 6, Dec. 1996, 32 pages. Certificate of Examination dated May 6, 2013 for Australian Patent Sycara, K., et al., “Dynamic Service Matchmaking Among Agents in No. 2012101472, 5 pages. OpenInformation Environments.” 1999, SIGMOD Record, 7 pages. Certificate of Examination dated May 6, 2013 for Australian Patent Sycara, K., et al., “The RETSINA MAS Infrastructure.” 2003, No. 2012101469, 4 pages. Autonomous Agents and Multi-Agent Systems, vol. 7, 20 pages. Certificate of Examination dated May 13, 2013 for Australian Patent Tyson, M., et al., “Domain-Independent Task Specification in the No. 2012101465, 5 pages. TACITUS Natural Language System.” May 1990, SRI International, Certificate of Examination dated May 13, 2013 for Australian Patent Artificial Intelligence Center, 16 pages. No. 2012101467, 5 pages. US 9,318.108 B2 Page 32

(56) References Cited Atal, B. S., “Efficient Coding of LPC Parameters by Temporal Decomposition.” IEEE International Conference on Acoustics, OTHER PUBLICATIONS Speech, and Signal Processing (ICASSP83), Apr. 1983, 4 pages. Bahl, L. R., et al., "Acoustic Markov Models Used in the Tangora Extended European Search Report dated Jul. 16, 2013, received in Speech Recognition System.” In Proceeding of International Confer Application No. 12186663.6-1910, which corresponds to U.S. Appl. ence on Acoustics, Speech, and Signal Processing (ICASSP88), Apr. No. 13/250,854, 8 pages (Gruber). 11-14, 1988, vol. 1, 4 pages. Notice of Allowance dated Jul. 10, 2013, received in U.S. Appl. No. Bahl, L. R. et al., “A Maximum Likelihood Approach to Continuous 13/725,656, 14 pages (Gruber). Speech Recognition.” IEEE Transaction on Pattern Analysis and Notice of Allowance dated Jun. 12, 2013, received in U.S. Appl. No. Machine Intelligence, vol. PAMI-5. No. 2, Mar. 1983, 13 pages. 11/518,292, 16 pages (Cheyer). Bahl, L. R. et al., “A Tree-Based Statistical Language Model for Final Office Action dated Jun. 13, 2013, received in U.S. Appl. No. Natural Language Speech Recognition.” IEEE Transactions on 13/251,118, 42 pages (Gruber). Acoustics, Speech and Signal Processing, vol. 37, Issue 7, Jul. 1989, Office Action dated Jul. 26, 2013, received in U.S. Appl. No. 8 pages. 13/725,512, 36 pages (Gruber). Bahl, L. R. et al., "Large Vocabulary Natural Language Continuous Office Action dated Jul. 11, 2013, received in U.S. Appl. No. Speech Recognition.” In Proceedings of 1989 International Confer 13/784,707, 29 pages (Cheyer). ence on Acoustics, Speech, and Signal Processing, May 23-26, 1989, Office Action dated Jul. 5, 2013, received in U.S. Appl. No. vol. 1, 6 pages. 13/725,713, 34 pages (Guzzoni). Bahl, L. R. etal. “Multonic Markov Word Models for Large Vocabu Office Action dated Jul. 2, 2013, received in U.S. Appl. No. lary Continuous Speech Recognition.” IEEE Transactions on Speech 13/725,761.14 pages (Gruber). and Audio Processing, vol. 1, No. 3, Jul. 1993, 11 pages. Office Action dated Jun. 28, 2013, received in U.S. Appl. No. Bahl, L. R. et al., “Speech Recognition with Continuous-Parameter 13/725,616, 29 pages (Cheyer). Hidden Markov Models.” In Proceeding of International Conference Office Action dated Jun. 27, 2013, received in U.S. Appl. No. on Acoustics, Speech, and Signal Processing (ICASSP88), Apr. 13/725,742, 29 pages (Cheyer). 11-14, 1988, vol. 1, 8 pages. Office Action dated May 23, 2013, received in U.S. Appl. No. Banbrook, M., “Nonlinear Analysis of Speech from a Synthesis 13/784,694, 27 pages (Gruber). Perspective.” A thesis submitted for the degree of Doctor of Philoso Office Action dated Jul. 5, 2013, received in U.S. Appl. No. phy, The University of Edinburgh, Oct. 15, 1996, 35 pages. 13/725,481, 26 pages (Gruber). Belaid, A., et al., “A Syntactic Approach for Handwritten Mathemati Australian Office Action dated Oct. 31, 2012 for Application No. cal Formula Recognition.” IEEE Transactions on Pattern Analysis 2012101 191, 6 pages. and Machine Intelligence, vol. PAMI-6, No. 1, Jan. 1984, 7 pages. Martin, D., et al., “The Open Agent Architecture: A Framework for Bellegarda, E. J., et al., "On-Line Handwriting Recognition Using building distributed software systems.” Jan.-Mar. 1999, Applied Arti Statistical Mixtures.” Advances in Handwriting and Drawings: A ficial Intelligence: An International Journal, vol. 13, No. 1-2, http:// Multidisciplinary Approach, Europia, 6th International IGS Confer adam.cheyer.com/papers/oaa.pdf, 38 pages. ence on Handwriting and Drawing, Paris-France, Jul. 1993, 11 pages. Acero, A., et al., “Environmental Robustness in Automatic Speech Bellegarda, J. R. “A Latent Semantic Analysis Framework for Large Recognition.” International Conference on Acoustics, Speech, and Span Language Modeling.' 5th European Conference on Speech, Signal Processing (ICASSP'90), Apr. 3-6, 1990, 4 pages. Communication and Technology, (Eurospeech'97), Sep. 22-25, Acero, A., et al., “Robust Speech Recognition by Normalization of 1997, 4 pages. the Acoustic Space.” International Conference on Acoustics, Speech, Bellegarda, J. R. “A Multispan Language Modeling Framework for and Signal Processing, 1991, 4 pages. Large Vocabulary Speech Recognition.” IEEE Transactions on Ahlbom, G., et al., “Modeling Spectral Speech Transitions. Using Speech and Audio Processing, vol. 6, No. 5, Sep. 1998, 12 pages. Temporal Decomposition Techniques.” IEEE International Confer Bellegarda, J. R., et al., “A Novel Word Clustering Algorithm Based ence of Acoustics, Speech, and Signal Processing (ICASSP87), Apr. on Latent Semantic Analysis.” In Proceedings of the IEEE Interna 1987, vol. 12, 4 pages. tional Conference on Acoustics, Speech, and Signal Processing Aikawa, K., “Speech Recognition Using Time-Warping Neural Net (ICASSP'96), vol. 1, 4 pages. works.” Proceedings of the 1991 IEEE Workshop on Neural Net Bellegarda, J. R., et al., “Experiments Using Data Augmentation for works for Signal Processing, Sep. 30 to Oct. 1, 1991, 10 pages. Speaker Adaptation.” International Conference on Acoustics, Anastasakos, A., et al., “Duration Modeling in Large Vocabulary Speech, and Signal Processing (ICASSP'95), May 9-12, 1995, 4 Speech Recognition.” International Conference on Acoustics, pageS. Speech, and Signal Processing (ICASSP'95), May 9-12, 1995, 4 Bellegarda, J. R. “Exploiting Both Local and Global Constraints for pageS. Multi-Span Statistical Language Modeling.” Proceeding of the 1998 Anderson, R. H., “Syntax-Directed Recognition of Hand-Printed IEEE International Conference on Acoustics, Speech, and Signal Two-Dimensional Mathematics.” In Proceedings of Symposium on Processing (ICASSP'98), vol. 2, May 12-15, 1998, 5 pages. Interactive Systems for Experimental Applied Mathematics: Pro Bellegarda, J. R. “Exploiting Latent Semantic Information in Statis ceedings of the Association for Computing Machinery Inc. Sympo tical Language Modeling.” In Proceedings of the IEEE, Aug. 2000, sium, (C) 1967. 12 pages. vol. 88, No. 8, 18 pages. Ansari, R., et al., “Pitch Modification of Speech using a Low-Sensi Bellegarda, J.R., “Interaction-Driven Speech Input—A Data-Driven tivity Inverse Filter Approach.” IEEE Signal Processing Letters, vol. Approach to the Capture of Both Local and Global Language Con 5. No. 3, Mar. 1998, 3 pages. straints.” 1992, 7 pages, available at http://old sigchi.org/bulletin? Anthony, N.J., et al., “Supervised Adaption for Signature Verification 1998.2/bellegarda.html. System.” Jun. 1, 1978, IBM Technical Disclosure, 3 pages. Bellegarda, J. R., "Large Vocabulary Speech Recognition with Apple Computer, "Guide Maker User's Guide.” (C) Apple Computer, Multispan Statistical Language Models.” IEEE Transactions on Inc., Apr. 27, 1994, 8 pages. Speech and Audio Processing, vol. 8, No. 1, Jan. 2000, 9 pages. Apple Computer, “Introduction to Apple Guide.” (C) Apple Computer, Bellegarda, J. R. et al., “Performance of the IBM Large Vocabulary Inc., Apr. 28, 1994, 20 pages. Continuous Speech Recognition System on the ARPA Wall Street Asanovi?, K., et al., “Experimental Determination of Precision Journal Task.” Signal Processing VII: Theories and Applications, (C) Requirements for Back-Propagation Training of Artificial Neural 1994 European Association for Signal Processing, 4 pages. Networks.” In Proceedings of the 2nd International Conference of Bellegarda, J. R. et al., “The Metamorphic Algorithm: A Speaker Microelectronics for Neural Networks, 1991, www.ICSI.Berkeley. Mapping Approach to Data Augmentation.” IEEE Transactions on EDU, 7 pages. Speech and Audio Processing, vol. 2, No. 3, Jul. 1994, 8 pages. US 9,318.108 B2 Page 33

(56) References Cited Hermansky, H., “Recognition of Speech in Additive and Convolu tional Noise Based on Rasta Spectral Processing.” In proceedings of OTHER PUBLICATIONS IEEE International Conference on Acoustics, speech, and Signal Processing (ICASSP93), Apr. 27-30, 1993, 4 pages. Black, A.W., et al., “Automatically Clustering Similar Units for Unit Hoehfeld M. et al., “Learning with Limited Numerical Precision Selection in Speech Synthesis.” In Proceedings of Eurospeech 1997. Using the Cascade-Correlation Algorithm.” IEEE Transactions on vol. 2, 4 pages. Neural Networks, vol. 3, No. 4, Jul. 1992, 18 pages. Blair, D.C., et al., “An Evaluation of Retrieval Effectiveness for a Holmes, J. N. "Speech Synthesis and Recognition—Stochastic Full-Text Document-Retrieval System.” Communications of the Models for Word Recognition.” Speech Synthesis and Recognition, ACM. vol. 28, No. 3, Mar. 1985, 11 pages. Briner, L. L., “Identifying Keywords in Text Data Processing.” In Published by Chapman & Hall, London, ISBN 0412534304, (C) 1998 Zelkowitz, Marvin V., ED, Directions and Challenges, 15th Annual J. N. Holmes, 7 pages. Technical Symposium, Jun. 17, 1976, Gaithersbury, Maryland, 7 Hon, H.W., et al., “CMU Robust Vocabulary-Independent Speech pageS. Recognition System.” IEEE International Conference on Acoustics, Bulyko, I., et al., "Joint Prosody Prediction and Unit Selection for Speech, and Signal Processing (ICASSP-91), Apr. 14-17, 1991, 4 Concatenative Speech Synthesis.” Electrical Engineering Depart pageS. ment, University of Washington, Seattle, 2001, 4 pages. IBM Technical Disclosure Bulletin, “Speech Editor.” vol. 29, No. 10, Bussey, H. E., et al., “Service Architecture, Prototype Description, Mar. 10, 1987, 3 pages. and Network Implications of a Personalized Information Grazing IBM Technical Disclosure Bulletin, “Integrated Audio-Graphics Service.” INFOCOM'90, Ninth Annual Joint Conference of the IEEE User Interface.” vol. 33, No. 11, Apr. 1991, 4 pages. Computer and Communication Societies, Jun. 3-7, 1990, http:// IBM Technical Disclosure Bulletin, “Speech Recognition with Hid slrohall.com/publications, 8 pages. den Markov Models of SpeechWaveforms.” vol.34, No. 1, Jun. 1991, Buzo, A., et al., “Speech Coding Based Upon Vector Quantization.” 10 pages. IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. Iowegian International, “FIR Filter Properties.” dispGuro, Digital Assp-28, No. 5, Oct. 1980, 13 pages. Signal Processing Central. http://www.dsaguru.com/dspitaqs firf Caminero-Gil. J., et al., “Data-Driven Discourse Modeling for properties, downloaded on Jul. 28, 2010, 6 pages. Semantic Interpretation.” In Proceedings of the IEEE International Jacobs, P. S., et al., "Scisor: Extracting Information from On-Line Conference on Acoustics, Speech, and Signal Processing, May 7-10, News. Communications of the ACM, vol. 33, No. 11, Nov. 1990, 10 1996, 6 pages. pageS. Cawley, G. C., “The Application of Neural Networks to Phonetic Jelinek, F. “Self-Organized Language Modeling for Speech Recog Modelling.” PhD Thesis, University of Essex, Mar. 1996, 13 pages. nition.” Readings in Speech Recognition, edited by Alex Waibel and Chang, S., et al., “A Segment-based Speech Recognition System for Kai-Fu Lee, May 15, 1990, (C) 1990 Morgan Kaufmann Publishers, Isolated Mandarin Syllables.” Proceedings TENCON 93, IEEE Inc., ISBN: 1-55860-124-4, 63 pages. Region 10 conference on Computer, Communication, Control and Jennings, A., et al., “A Personal News Service Based on a User Model Power Engineering, Oct. 19-21, 1993, vol. 3, 6 pages. Neural Network.” IEICE Transactions on Information and Systems, Conklin, J., “Hypertext: An Introduction and Survey.” Computer vol. E75-D, No. 2, Mar. 1992, Tokyo, JP, 12 pages. Magazine, Sep. 1987. 25 pages. Ji, T., et al., “A Method for Chinese Syllables Recognition based upon Connolly, F.T. et al., “Fast Algorithms for Complex Matrix Multi Sub-syllable Hidden Markov Model.” 1994 International Sympo plication Using Surrogates.” IEEE Transactions on Acoustics, sium on Speech, Image Processing and Neural Networks, Apr. 13-16, Speech, and Signal Processing, Jun. 1989, vol. 37, No. 6, 13 pages. 1994, Hong Kong, 4 pages. Deerwester, S., et al., “Indexing by Latent Semantic Analysis,” Jour Jones, J., “Speech Recognition for Cyclone.' Apple Computer, Inc., nal of the American Society for Information Science, vol. 41, No. 6, E.R.S., Revision 2.9, Sep. 10, 1992, 93 pages. Sep. 1990, 19 pages. Katz, S. M.. “Estimation of Probabilities from Sparse Data for the Deller, Jr., J.R., et al., “Discrete-Time Processing of Speech Signals.” Language Model Component of a Speech Recognizer.” IEEE Trans (C) 1987 Prentice Hall, ISBN: 0-02-328301-7, 14 pages. actions on Acoustics, Speech, and Signal Processing, vol. ASSP-35, Digital Equipment Corporation, “OpenVMS Software Overview.” No. 3, Mar. 1987, 3 pages. Dec. 1995, software manual, 159 pages. Kitano, H., “PhilDM-Dialog. An Experimental Speech-to-Speech Donovan, R. E., “A New Distance Measure for Costing Spectral Dialog Translation System.” Jun. 1991 Computer, vol. 24. No. 6, 13 Discontinuities in Concatenative Speech Synthesisers.” 2001, http:// pageS. citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.21.6398, 4 Klabbers, E., et al., “Reducing Audible Spectral Discontinuities.” pageS. IEEE Transactions on Speech and Audio Processing, vol. 9, No. 1, Frisse, M. E., "Searching for Information in a Hypertext Medical Jan. 2001, 13 pages. Handbook.” Communications of the ACM, vol. 31, No. 7, Jul. 1988, Klatt, D. H., "Linguistic Uses of Segmental Duration in English: 8 pages. Acoustic and Perpetual Evidence.” Journal of the Acoustical Society Goldberg, D., et al., “Using Collaborative Filtering to Weave an of America, vol. 59, No. 5, May 1976, 16 pages. Information Tapestry.” Communications of the ACM, vol.35, No. 12, Kominek, J., et al., “Impact of Durational Outlier Removal from Unit Dec. 1992, 10 pages. Selection Catalogs,” 5th ISCA Speech Synthesis Workshop, Jun. Gorin, A. L., et al., “On Adaptive Acquisition of Language.” Inter 14-16, 2004, 6 pages. national Conference on Acoustics, Speech, and Signal Processing Kubala, F., et al., “Speaker Adaptation from a Speaker-Independent (ICASSP90), vol. 1, Apr. 3-6, 1990, 5 pages. Training Corpus.” International Conference on Acoustics, Speech, Gotoh, Y., et al., “Document Space Models Using Latent Semantic and Signal Processing (ICASSP'90), Apr. 3-6, 1990, 4 pages. Analysis.” In Proceedings of Eurospeech, 1997, 4 pages. Kubala, F., et al., “The Hub and Spoke Paradigm for CSR Evalua Gray, R. M., “Vector Quantization.” IEEE ASSP Magazine, Apr. tion.” Proceedings of the Spoken Language Technology Workshop, 1984, 26 pages. Mar. 6-8, 1994.9 pages. Harris, F.J., “On the Use of Windows for Harmonic Analysis with the Lee, K.F., "Large-Vocabulary Speaker-Independent Continuous Discrete Fourier Transform.” In Proceedings of the IEEE, vol. 66, No. Speech Recognition: The SPHINX System.” Apr. 18, 1988, Partial 1, Jan. 1978, 34 pages. fulfillment of the requirements for the degree of Doctor of Philoso Helm, R., et al., “Building Visual Language Parsers.” In Proceedings phy, Computer Science Department, Carnegie Mellon University, ofCHI’91 Proceedings of the SIGCHI Conference on Human Factors 195 pages. in Computing Systems, 8 pages, 1991. Lee, L., et al., “A Real-Time Mandarin Dictation Machine for Chi Hermansky, H., “Perceptual Linear Predictive (PLP) Analysis of nese Language with Unlimited Texts and Very Large Vocabulary.” Speech.” Journal of the Acoustical Society of America, vol. 87, No. 4. International Conference on Acoustics, Speech and Signal Process Apr. 1990, 15 pages. ing, vol. 1, Apr. 3-6, 1990, 5 pages. US 9,318.108 B2 Page 34

(56) References Cited Ratcliffe, M.. “Clear Access 2.0 allows SQL searches off-line.” (Structured Query Language), Clear Acess Corp., MacWeek Nov. 16. OTHER PUBLICATIONS 1992, vol. 6, No. 41, 2 pages. Remde, J.R., et al., “SuperBook: An Automatic Tool for Information Lee, L, et al., “Golden Mandarin(II)—An Improved Single-Chip Exploration-Hypertext?.” In Proceedings of Hypertext'87 papers, Real-Time Mandarin Dictation Machine for Chinese Language with Nov. 13-15, 1987, 14 pages. Very Large Vocabulary.” 0-7803-0946-4/93 (C) 1993 IEEE, 4 pages. Reynolds, C. F., "On-Line Reviews: A New Application of the Lee, L, et al., “Golden Mandarin(II)—An Intelligent Mandarin Dic HICOM Conferencing System.” IEE Colloquium on Human Factors tation Machine for Chinese Character Input with Adaptation/Learn in Electronic Mail and Conferencing Systems, Feb. 3, 1989, 4 pages. ing Functions.” International Symposium on Speech, Image Process Rigoll, G., “Speaker Adaptation for Large Vocabulary Speech Rec ing and Neural Networks, Apr. 13-16, 1994, Hong Kong, 5 pages. Lee, L., et al., “System Description of Golden Mandarin (I) Voice ognition Systems. Using Speaker Markov Models.” International Input for Unlimited Chinese Characters.” International Conference Conference on Acoustics, Speech, and Signal Processing on Computer Processing of Chinese & Oriental Languages, vol. 5, (ICASSP'89), May 23-26, 1989, 4 pages. Nos. 3 & 4. Nov. 1991, 16 pages. Riley, M.D., "Tree-Based Modelling of Segmental Durations.” Talk Lin, C.H., et al., “A New Framework for Recognition of Mandarin ing Machines Theories, Models, and Designs, 1992 C. Elsevier Sci Syllables WithTones Using Sub-syllabic Unites.” IEEE International ence Publishers B.V., North-Holland, ISBN: 08-444-89115.3, 15 Conference on Acoustics, Speech, and Signal Processing (ICASSP pageS. 93), Apr. 27-30, 1993, 4 pages. Rivoira, S., et al., “Syntax and Semantics in a Word-Sequence Rec Linde, Y., et al., “An Algorithm for Vector Quantizer Design.” IEEE ognition System.” IEEE International Conference on Acoustics, Transactions on Communications, vol. 28, No. 1, Jan. 1980, 12 pages. Speech, and Signal Processing (ICASSP79), Apr. 1979, 5 pages. Liu, F.H., et al., “Efficient Joint Compensation of Speech for the Rosenfeld, R., "A Maximum Entropy Approach to Adaptive Statis Effects of Additive Noise and Linear Filtering.” IEEE International tical Language Modelling.” Computer Speech and Language, vol. 10, Conference of Acoustics, Speech, and Signal Processing, ICASSP No. 3, Jul. 1996, 25 pages. 92, Mar. 23-26, 1992, 4 pages. Roszkiewicz, A., “Extending your Apple.” BackTalk Lip Service, Logan, B., “Mel Frequency Cepstral Coefficients for Music Model A+ Magazine, The Independent Guide for Apple Computing, vol. 2, ing.” In International Symposium on Music Information Retrieval, No. 2, Feb. 1984, 5 pages. 2000, 2 pages. Sakoe, H., et al., “Dynamic Programming Algorithm Optimization Lowerre, B.T., "The-Harpy Speech Recognition System.” Doctoral Dissertation, Department of Computer Science, Carnegie Mellon for Spoken Word Recognition.” IEEE Transactins on Acoustics, University, Apr. 1976, 20 pages. Speech, and Signal Processing, Feb. 1978, vol. ASSP-26 No. 1, 8 Maghbouleh, A., “An Empirical Comparison of Automatic Decision pageS. Tree and Linear Regression Models for Vowel Durations.” Revised Salton, G., et al., “On the Application of Syntactic Methodologies in version of a paper presented at the Computational Phonology in Automatic Text Analysis.” Information Processing and Management, Speech Technology workshop, 1996 annual meeting of the Associa vol. 26, No. 1, Great Britain 1990, 22 pages. tion for Computational Linguistics in Santa Cruz, California, 7 pages. Savoy, J., “Searching Information in Hypertext Systems. Using Mul Markel, J. D., et al., “Linear Prediction of Speech.” Springer-Verlag, tiple Sources of Evidence.” International Journal of Man-Machine Berlin Heidelberg New York 1976, 12 pages. Studies, vol. 38, No. 6, Jun. 1993, 15 pages. Morgan, B., “Business Objects.” (Business Objects for Windows) Scagliola, C., "Language Models and Search Algorithms for Real Business Objects Inc., DBMS Sep. 1992, vol. 5, No. 10, 3 pages. Time Speech Recognition.” International Journal of Man-Machine Mountford, S.J., et al., “Talking and Listening to Computers.” The Studies, vol. 22, No. 5, 1985, 25 pages. Art of Human-Computer Interface Design, Copyright (C) 1990 Apple Schmandt, C., et al., “Augmenting a Window System with Speech Computer, Inc. Addison-Wesley Publishing Company, Inc., 17 pages. Input.” IEEE Computer Society, Computer Aug. 1990, vol. 23, No. 8, Murty, K. S. R., et al., "Combining Evidence from Residual Phase 8 pages. and MFCC Features for Speaker Recognition.” IEEE Signal Process Schütze, H., “Dimensions of Meaning.” Proceedings of ing Letters, vol. 13, No. 1, Jan. 2006, 4 pages. Supercomputing'92 Conference, Nov. 16-20, 1992, 10 pages. Murveit H. et al., “Integrating Natural Language Constraints into Sheth B., et al., “Evolving Agents for Personalized Information Fil HMM-based Speech Recognition.” 1990 International Conference tering.” In Proceedings of the Ninth Conference on Artificial Intelli on Acoustics, Speech, and Signal Processing, Apr. 3-6, 1990, 5 pages. gence for Applications, Mar. 1-5, 1993, 9 pages. Nakagawa, S., et al., “Speaker Recognition by Combining MFCC Shikano, K., et al., “Speaker Adaptation Through Vector Quantiza and Phase Information.” IEEE International Conference on Acoustics tion.” IEEE International Conference on Acoustics, Speech, and Sig Speech and Signal Processing (ICASSP), Mar. 14-19, 2010, 4 pages. nal Processing (ICASSP86), vol. 11, Apr. 1986, 4 pages. Niesler, T. R. et al., “A Variable-Length Category-Based N-Gram Sigurdsson, S., et al., “Mel Frequency Cepstral Coefficients: An Language Model.” IEEE International Conference on Acoustics, Evaluation of Robustness of MP3 Encoded Music.” In Proceedings of Speech, and Signal Processing (ICASSP'96), vol. 1, May 7-10, 1996, the 7th International Conference on Music Information Retrieval 6 pages. (ISMIR), 2006, 4 pages. Papadimitriou, C. H., et al., “Latent Semantic Indexing: A Probabi Silverman, K. E. A., et al., “Using a Sigmoid Transformation for listic Analysis.” Nov. 14, 1997, http://citeseerx.ist.psu.edu/messages/ Improved Modeling of Phoneme Duration.” Proceedings of the IEEE downloadsexceeded.html, 21 pages. International Conference on Acoustics, Speech, and Signal Process Parsons, T.W. “Voice and Speech Processing.” Linguistics and Tech ing, Mar. 15-19, 1999, 5 pages. nical Fundamentals, Articulatory Phonetics and Phonemics, (C) 1987 Tenenbaum, A.M., et al., “Data Structure Using Pascal.” 1981 McGraw-Hill, Inc., ISBN: 0-07-0485541-0, 5 pages. Prentice-Hall, Inc., 34 pages. Parsons, T. W. “Voice and Speech Processing.” Pitch and Formant Tsai, W.H., et al., “Attributed Grammar—A Tool for Combining Estimation, (C) 1987 McGraw-Hill, Inc., ISBN: 0-07-0485541-0, 15 Syntactic and Statistical Approaches to Pattern Recognition.” IEEE pageS. Transactions on Systems, Man, and Cybernetics, vol. SMC-10, No. Picone, J., “Continuous Speech Recognition Using Hidden Markov 12, Dec. 1980, 13 pages. Models.” IEEE ASSP Magazine, vol. 7, No. 3, Jul 1990, 16 pages. Udell, J., “Computer Telephony.” BYTE, vol. 19, No. 7, Jul. 1, 1994, Rabiner, L. R., et al., “Fundamental of Speech Recognition.” (C) 1993 9 pages. AT&T. Published by Prentice-Hall, Inc., ISBN: 0-13-285.826-6, 17 van Santen, J. P. H., "Contextual Effects on Vowel Duration.” Journal pageS. Speech Communication, vol. 11, No. 6, Dec. 1992, 34 pages. Rabiner, L. R. et al., “Note on the Properties of a Vector Quantizer for Vepa, J., et al., “New Objective Distance Measures for Spectral LPCCoefficients.” The Bell SystemTechnical Journal, vol. 62, No. 8, Discontinuities in Concatenative Speech Synthesis.” In Proceedings Oct. 1983, 9 pages. of the IEEE 2002 Workshop on Speech Synthesis, 4 pages. US 9,318.108 B2 Page 35

(56) References Cited Gong, J., et al., “Guidelines for Handheld Mobile Device Interface Design.” Proceedings of DSI 2004 Annual Meeting, pp. 3751-3756. OTHER PUBLICATIONS Horvitz, E., "Handsfree Decision Support: Toward a Non-invasive Human-Computer Interface.” Proceedings of the Symposium on Verschelde, J., “MATLAB Lecture 8. Special Matrices in Computer Applications in Medical Care, IEEE Computer Society MATLAB.” Nov. 23, 2005. UIC Dept. of Math. Stat. & C.S., MCS Press, Nov. 1995, 1 page. 320, Introduction to Symbolic Computation, 4 pages. Horvitz, E., “In Pursuit of Effective Handsfree Decision Support: Vingron, M. "Near-Optimal Sequence Alignment.” Deutsches Coupling Bayesian Inference, Speech Understanding, and User Krebsforschungszentrum (DKFZ), Abteilung Theoretische Models.” 1995, 8 pages. Bioinformatik, Heidelberg, Germany, Jun. 1996, 20 pages. “Top 10 Best Practices for Voice User Interface Design.” Nov. 1, Werner, S., et al., “Prosodic Aspects of Speech.” Université de 2002, http://www.developer.com/voice/article.php/1567051/Top Lausanne, Switzerland, 1994, Fundamentals of Speech Synthesis 10-Best-Practices-for-Voice-User-Interface-Design.htm, 4 pages. and Speech Recognition: Basic Concepts, State of the Art, and Future Australian Office Action dated Dec. 7, 2012 for Application No. Challenges, 18 pages. 2010254812, 8 pages. Wolff, M., “Poststructuralism and the Artful Database: Some Theo Australian Office Action dated Nov. 27, 2012 for Application No. retical Considerations.” Information Technology and Libraries, vol. 2012101471, 6 pages. 13, No. 1, Mar. 1994, 10 pages. Australian Office Action dated Nov. 22, 2012 for Application No. Wu, M., “Digital Speech Processing and Coding.” ENEE408G 2012101466, 6 pages. Capstone-Multimedia Signal Processing, Spring 2003, Lecture-2 Australian Office Action dated Nov. 14, 2012 for Application No. course presentation, University of Maryland, College Park, 8 pages. 2012101473, 6 pages. Wu, M., “Speech Recognition, Synthesis, and H.C.I.” ENEE408G Australian Office Action dated Nov. 19, 2012 for Application No. Capstone-Multimedia Signal Processing, Spring 2003, Lecture-3 2012101470, 5 pages. course presentation, University of Maryland, College Park, 11 pages. Australian Office Action dated Nov. 28, 2012 for Application No. Wyle, M. F., “A Wide Area Network Information Filter.” In Proceed 2012101468, 5 pages. ings of First International Conference on Artificial Intelligence on Australian Office Action dated Nov. 19, 2012 for Application No. Wall Street, Oct. 9-11, 1991, 6 pages. 2012101472, 5 pages. Yankelovich, N., et al., “Intermedia: The Concept and the Construc Australian Office Action dated Nov. 19, 2012 for Application No. tion of a Seamless Information Environment.” Computer Magazine, 2012101469, 6 pages. Jan. 1988, (C) 1988 IEEE, 16 pages. Australian Office Action dated Nov. 15, 2012 for Application No. Yoon, K., et al., “Letter-to-Sound Rules for Korean.” Department of 2012101465, 6 pages. Linguistics, The Ohio State University, 2002, 4 pages. Australian Office Action dated Nov. 30, 2012 for Application No. Zhao, Y., “An Acoustic-Phonetic-Based Speaker Adaptation Tech 2012101467, 6 pages. nique for Improving Speaker-Independent Continuous Speech Rec Canadian Office Action dated Mar. 27, 2013 for Application No. ognition.” IEEE Transactions on Speech and Audio Processing, vol. 2.793,118, 3 pages. 2, No. 3, Jul. 1994, 15 pages. Current claims of PCT Application No. PCT/US 11/20861 dated Jan. Zovato, E., et al., “Towards Emotional Speech Synthesis: A Rule 11, 2011, 17 pages. Based Approach. 2 pages. Notice of Allowance dated Feb. 29, 2012, received in U.S. Appl. No. International Search Report dated Nov. 9, 1994, received in Interna 11/518,292, 29 pages (Cheyer). tional Application No. PCT/US 1993/12666, which corresponds to Final Office Action dated Jun. 19, 2012, received in U.S. Appl. No. U.S. Appl. No. 07/999,302, 8 pages (Robert Don Strong). 12479,477. 46 pages (van Os). International Preliminary Examination Report dated Mar. 1, 1995, Final Office Action dated May 10, 2011, received in U.S. Appl. No. received in International Application No. PCT/US 1993/12666, 11/518,292, 14 pages (Cheyer). which corresponds to U.S. Appl. No. 07/999,302, 5 pages (Robert Final Office Action dated Mar. 25, 2013, received in U.S. Appl. No. Don Strong). 13/251,127. 53 pages (Gruber). International Preliminary Examination Report dated Apr. 10, 1995, Office Action dated Sep. 29, 2011, received in U.S. Appl. No. received in International Application No. PCT/US 1993/12637, 12479,477, 32 pages (van Os). which corresponds to U.S. Appl. No. 07/999,354, 7 pages (Alejandro Office Action dated Nov. 24, 2010, received in U.S. Appl. No. Acero). 11/518,292, 12 pages (Cheyer). International Search Report dated Feb. 8, 1995, received in Interna Office Action dated Nov. 9, 2009, received in U.S. Appl. No. tional Application No. PCT/US 1994/11011, which corresponds to 11/518,292, 10 pages (Cheyer). U.S. Appl. No. 08/129,679, 7 pages (Yen-Lu Chow). Office Action dated Jan. 31, 2013, received in U.S. Appl. No. International Preliminary Examination Report dated Feb. 28, 1996, 13/251,088, 38 pages (Gruber). received in International Application No. PCT/US 1994/11011, Office Action dated Nov. 28, 2012, received in U.S. Appl. No. which corresponds to U.S. Appl. No. 08/129,679, 4 pages (Yen-Lu 13/251,104, 49 pages (Gruber). Chow). Office Action dated Dec. 7, 2012, received in U.S. Appl. No. Written Opinion dated Aug. 21, 1995, received in International 13/251,118, 52 pages (Gruber). Application No. PCT/US 1994/11011, which corresponds to U.S. Office Action dated Nov. 8, 2012, received in U.S. Appl. No. Appl. No. 08/129,679, 4 pages (Yen-Lu Chow). 13/251.127, 35 pages (Gruber). International Search Report dated Nov. 8, 1995, received in Interna Office Action dated Apr. 16, 2013, received in U.S. Appl. No. tional Application No. PCT/US 1995/08369, which corresponds to 13/725,550, 8 pages (Cheyer). U.S. Appl. No. 08/271,639, 6 pages (Peter V. De Souza). Office Action dated Mar. 27, 2013, received in U.S. Appl. No. International Preliminary Examination Report dated Oct. 9, 1996, 13/725,656, 22 pages (Gruber). received in International Application No. PCT/US 1995/08369, Russian Office Action dated Nov. 8, 2012 for Application No. which corresponds to U.S. Appl. No. 08/271,639, 4 pages (Peter V. 2012144647, 7 pages. De Souza). Russian Office Action dated Dec. 6, 2012 for Application No. Office Action dated Mar. 7, 2013, received in U.S. Appl. No. 2012144605, 6 pages. 13/492,809, 26 pages (Gruber). GB Patent Act 1977: Combined Search Report and Examination Car Working Group, “Bluetooth Doc Hands-Free Profile 1.5 HFP1. Report under Sections 17 and 18(3) for Application No. GB10093.18. 5 SPEC.” Nov. 25, 2005. www.bluetooth.org, 84 pages. 5, report dated Oct. 8, 2010, 5 pages. Cohen, Michael H., et al., “Voice User Interface Design.” excerpts GB Patent Act 1977: Combined Search Report and Examination from Chapter 1 and Chapter 10, Addison-Wesley ISBN: 0-321 Report under Sections 17 and 18(3) for Application No. GB1217449. 18576-5, 2004, 36 pages. 6, report dated Jan. 17, 2013, 6 pages. US 9,318.108 B2 Page 36

(56) References Cited 1996, as depicted in Exemplary Screenshots from video entitled Demonstration Video of Multimodal Maps Using an Agent Archi OTHER PUBLICATIONS tecture', 1996, 6 pages. Cheyer et al., “Demonstration Video of Multimodal Maps Using an International Search Report and Written Opinion dated Aug. 25. Open-Agent Architecture', published by SRI International no later 2010, received in International Application No. PCT/US2010/ than 1996, as depicted in Exemplary Screenshots from video entitled 037378, which corresponds to U.S. Appl. No. 12/479,477, 16 pages Demonstration Video of Multimodal Maps Using an Open-Agent (Apple Inc.). Architecture, 6 pages. International Search Report and Written Opinion dated Nov. 16. Cheyer, A., “Demonstration Video of Vanguard Mobile Portal', pub 2012, received in International Application No. PCT/US2012/ lished by SRI International no later than 2004, as depicted in Exem 040571, which corresponds to U.S. Appl. No. 13/251,088 14 pages plary Screenshots from video entitled Demonstration Video of Van (Apple Inc.). guard Mobile Portal, 2004, 10 pages. International Search Report and Written Opinion dated Dec. 20. Choi et al., "Acoustic and Visual Signal based Context Awareness 2012, received in International Application No. PCT/US2012/ System for Mobile Application', IEEE Transactions on Consumer Electronics, vol. 57, No. 2, May 2011, pp. 738-746. 056382, which corresponds to U.S. Appl. No. 13/250.947, 11 pages Guzzoni et al., “Modeling Human-Agent Interaction with Active (Gruber). Ontologies'. AAAI Spring Symposium, Interaction Challenges for Extended European Search Report received for European Patent Intelligent Assistants, Stanford University, Palo Alto, California, Application No. 12185276.8, mailed on Dec. 18, 2012, 4 pages. 2007, 8 pages. International Preliminary Report on Patentability received for PCT Kickstarter, "Ivee Sleek: Wi-Fi Voice-Activated Assistant', available Patent Application No. PCT/US2012/029810, mailed on Oct. 3, online at shttps://www.kickstarter.com/projectsfivee?ivee-sleek-wi 2013, 9 pages. fi-voice-activated-assistant retrived from intemet on Feb. 10, International Search Report and Written Opinion received for PCT 2014, 13 pages. Patent Application No. PCT/US2012/029810, mailed on Aug. 17, Navigli, Roberto, “Word Sense Disambiguation: A Survey', Article 2012, 11 pages. 10, ACM Computing Surveys, vol. 41, No. 2, Feb. 2009, 70 pages. International Search Report and Written Opinion received for PCT Parson, T. W. “Voice and Speech Processing”. Pitch and Formant Patent Application No. PCT/US2013/040971, mailed on Nov. 12, Estimation (C) 1987, McGraw-Hill, Inc., ISBN: 0-07-0485541-0, 2013, 11 pages. 1987, 15 pages. “2004 Chrysler Pacifica: U-Connect Hands-Free Communication Vlingo, "Vlingo Launches Voice Enablement Application on Apple System'. The Best and Brightest of 2004, Brief Article, Automotive App Store'. Press Release, Dec. 3, 2008, 2 pages. Industries, Sep. 2003. Xu, "Speech-Based Interactive Games for Language Learning: “2007 Lexus GS 450h 4dr Sedan (3.5L 6cyl Gas/Electric Hybrid Reading, Translation, and Question-Answering'. Computational CVT)', available online at , Retrieved on Aug. 3, 2006. 2009, pp. 133-160. "All Music Website', available online at , Retrieved on Mar. 19, 2007. Riders, Aug. 22, 2002, 11 pages. “BluePhoneElite: About', available online at , Retrieved on Sep. 25, 2006. Conferencing'. Sunmicrosystems Laboratories, Dec. 1992, pp. 1-15. “BluePhoneElite: Features', available online at < http://www. Penn et al., "Ale for Speech: A Translation Prototype', Bell Labora reelintelligence.com/BluePhoneElite/features.shtml>, Retrieved on tories, 1999, 4 pages. Sep. 25, 2006. Phillipps, Ben, “Touchscreens are Changing the Face of Comput "Chrysler Pacifica: U-Connect Hands-Free Communication Sys ers Today's Users Have Five Types of Touchscreens to Choose tem', available online at retrived from internet on Feb. 10, 2014, 2 pages. Panel Proceedings, 1989, pp. 93-109. "Meet IveeYour Wi-Fi Voice Activated Assistant', available online at Pickering, J. A., “Touch-Sensitive Screens: The Technologies and retrived from internet on Feb. 10, Their Application'. International Journal of Man-Machine Studies, 2014, 8 pages. vol. 25, No. 3, Sep. 1986, pp. 249-269. “What is Fuzzy Logic?', available online at , Retrieved on Apr. 15, 1993. ACM Multimedia, Oct. 1999, pp. 373-382. “Speaker Recognition'. Wikipedia, The Free Enclyclopedia, Nov. 2, Plaisant et al., “Touchscreen Interfaces for Alphanumeric Data 2010, 3 pages. Entry”. Proceedings of the Human Factors and Ergonomics Society Apple Computer, Video Entitled, “Knowledge Navigator', published 36th Annual Meeting, 1992, pp. 293-297. by Apple Computer no later than 2008, as depicted in Exemplary Plaisant et al., “Touchscreen Toggle Design”. CHI'92, May 3-7, Screenshots from video entitled 'Knowledge Navigator', 2008, 7 1992, pp. 667-668. pageS. Poly-Optical Products, Inc., “Poly-Optical Fiber Optic Membrane Applebaum et al., “Enhancing the Discrimination of Speaker Inde Switch Backlighting, available at

(56) References Cited Roucos et al., “High Quality Time-Scale Modification for Speech'. Proceedings of the 1985 IEEE Conference on Acoustics, Speech and OTHER PUBLICATIONS Signal Processing, 1985, pp. 493-496. Sabin et al., “Product Code Vector Quantizers for Waveform and Rabiner et al., “Digital Processing of Speech Signals'. Prentice Hall, Voice Coding”. (IEEE Transactions on Acoustics, Speech and Signal 1978, pp. 274-277. Processing, Jun. 1984), as reprinted in Vector Quantization (IEEE Rampe et al., “SmartForm Designer and SmartForm Assistant”. Press, 1990), 1990, pp. 274-288. News release, Claris Corp., Jan. 9, 1989, 1 page. Santaholma, Marianne E. "Grammar Sharing Techniques for Rule Rao et al., “Exploring Large Tables with the Table Lens'. Apple Inc., based Multilingual NLP Systems'. Proceedings of the 16th Nordic Video Clip, Xerox Corp., on a CD, 1994. Conference of Computational Linguistics, NODALIDA 2007, May Rao et al., “Exploring Large Tables with the Table Lens', CHI’95 Mosaic of Creativity, ACM, May 7-11, 1995, pp. 403-404. 25, 2007, 8 pages. Rao et al., “The Table Lens: Merging Graphical and Symbolic Rep Santen, Jan P. "Assignment of Segmental Duration in Text-to resentations in an Interactive Focus+Context Visualization for Tabu Speech Synthesis'. Computer Speech and Language, vol. 8, No. 2, lar Information'. Proceedings of the ACM SIGCHI Conference on Apr. 1994, pp. 95-128. Human Factors in Computing Systems, Apr. 1994, pp. 1-7. Sarawagi, Sunita, “CRF Package Page', available at , retrieved on Apr. 6, 2011, 2 pages. 051), Dec. 1986, pp. 1-30. Sarkar et al., “Graphical Fisheye Views”. Communications of the Ratcliffe et al., “Intelligent Agents Take U.S. Bows'. MacWeek, vol. ACM, vol. 37, No. 12, Dec. 1994, pp. 73-83. 6, No. 9, Mar. 2, 1992, 1 page. Sarkar et al., “Graphical Fisheye Views of Graphs”. Systems Reddy, D. R. “Speech Recognition by Machine: A Review'. Pro Research Center, Digital Equipment Corporation Mar. 17, 1992, 31 ceedings of the IEEE, Apr. 1976, pp. 501-531. pageS. Reininger et al., “Speech and Speaker Independent Codebook Design Sarkar et al., “Graphical Fisheye Views of Graphs”. CHI '92 Pro in VQ Coding Schemes'. (Proceedings of the IEEE International ceedings of the SIGCHI Conference on Human Factors in Computing Acoustics, Speech and Signal Processing Conference, Mar. 1985), as Systems, May 3-7, 1992, pp. 83-91. reprinted in Vector Quantization (IEEE Press, 1990), 1990, pp. 271 Sarkar et al., “Stretching the Rubber Sheet: A Metaphor for Viewing 273. Large Layouts on Small Screens”, UIST'93, ACM, Nov. 3-5, 1993, Ren et al., “Efficient Strategies for Selecting Small Targets on Pen Based Systems: An Evaluation Experiment for Selection Strategies pp. 81-91. and Strategy Classifications'. Proceedings of the IFIP TC2/TC13 Sastry, Ravindra W. “A Need for Speed: A New Speedometer for WG2.7/WG 13.4 Seventh Working Conference on Engineering for Runners'. Submitted to the Department of Electrical Engineering and Human-Computer Interaction, vol. 150, 1998, pp. 19-37. Computer Science at the Massachusetts Institute of Technology, Ren et al., “Improving Selection Performance on Pen-Based Sys 1999, pp. 1-42. tems: A Study of Pen-Based Interaction for Selection Tasks'. ACM Schafer et al., “Digital Representations of Speech Signals'. Proceed Transactions on Computer-Human Interaction, vol. 7. No. 3, Sep. ings of the IEEE, vol. 63, No. 4, Apr. 1975, pp. 662-677. 2000, pp. 384-416. Schaffer et al., “Navigating Hierarchically Clustered Networks Ren et al., “The Best among Six Strategies for Selecting a Minute through Fisheye and Full-Zoom Methods'. ACM Transactions on Target and the Determination of the Minute Maximum Size of the Computer-Human Interaction, vol. 3, No. 2, Jun. 1996, pp. 162-188. Targets on a Pen-Based Computer'. Human-Computer Interaction Sullivan, Danny, “How Google Instant's Autocomplete Suggestions INTERACT, 1997, pp. 85-92. Work', available at (http://searchengineland.com/how-google-in Ricker, Thomas, "Apple Patents Audio User Interface', Engadget, stant-autocomplete-suggestions-work-62592, Apr. 6, 2011, 12 available at , May 4, 2006, 6 pages. Summerfield et al., “ASIC Implementation of the Lyon Cochlea Riecken, RD., "Adaptive Direct Manipulation', IEEE Xplore, 1991, Model”. Proceedings of the 1992 International Conference on Acous pp. 1115-1120. tics, Speech and Signal Processing, IEEE, vol. V, 1992, pp. 673-676. Rioport, “Rio 500: Getting Started Guide', available at mp3 player/creative MANUAL000023453.pdf>, 1999, 2 pages. muvo tx 256mb>, Aug. 17, 2004. 1 page. Robbin et al., “MP3 Player and Encoder for Macintosh!”. SoundJam Taos, "Taos, Inc. Announces Industry's First Ambient Light Sensor to MP Plus, Version 2.0, 2000, 76 pages. Convert Light Intensity to Digital Signals'. News Release, available Robertson et al., “Information Visualization Using 3D Interactive at , Sep. 16. Animation'. Communications of the ACM. vol. 36, No. 4, Apr. 1993, 2002, 3 pages. pp. 57-71. Taylor et al., “Speech Synthesis. By Phonological Structure Match Robertson et al., “The Document Lens', UIST 93, Nov. 3-5, 1993, ing. International Speech Communication Association, vol. 2, Sec pp. 101-108. tion 3, 1999, 4 pages. Root, Robert, “Design of a Multi-Media Vehicle for Social Brows Tello, Ernest R. “Natural-Language Systems', Mastering AI Tools ing', Bell Communications Research, 1988, pp. 25-38. and Techniques, Howard W. Sams & Company, 1988. Roseberry, Catherine, “How to Pair a Bluetooth Headset & Cell TG3 Electronics, Inc., “BL82 Series Backlit Keyboards', available at Phone', available at . usingyourphone/ht/blueheadset p.htm>, retrieved on Apr. 29, 2006, retrieved on Dec. 19, 2002, 2 pages. 2 pages. The HP 150, “Hardware: Compact, Powerful, and Innovative”, vol. 8, Rosenberg et al., “An Overview of the Andrew Message System'. No. 10, Oct. 1983, pp. 36-50. Information Technology Center Carnegie-Mellon University, Jul. Tidwell, Jenifer, "Animated Transition”. Designing Interfaces, Pat 1987, pp. 99-108. terns for effective Interaction Design, Nov. 2005, First Edition, 4 Rosner et al., “In Touch: A Graphical User Interface Development pageS. Tool”, IEEE Colloquium on Software Tools for Interface Design, Touch, Joseph, "Zoned Analog Personal Teleconferencing'. USC / Nov. 8, 1990, pp. 1211-1217. Information Sciences Institute, 1993, pp. 1-19. Rossfrank, "Konstenlose Sprachmitteilungins Festnetz'. Toutanova et al., “Feature-Rich Part-of-Speech Tagging with a XP002234425, Dec. 10, 2000, pp. 1-4. Cyclic Dependency Network”. Computer Science Dept., Stanford Roucos et al., “A Segment Vocoder at 150 B/S'. (Proceedings of the University, Stanford CA 94305-9040, 2003, 8 pages. IEEE International Acoustics, Speech and Signal Processing Confer Trigg et al., “Hypertext Habitats: Experiences of Writers in ence, Apr. 1983), as reprinted in Vector Quantization (IEEE Press, NoteCards'. Hypertext 87 Papers; Intelligent Systems Laboratory, 1990), 1990, pp. 246-249. Xerox Palo Alto Research Center, 1987, pp. 89-108. US 9,318.108 B2 Page 38

(56) References Cited Wong et al., “Very Low Data Rate Speech Compression with LPC Vector and Matrix Quantization'. (Proceedings of the IEEE Int’l OTHER PUBLICATIONS Acoustics, Speech and Signal Processing Conference, Apr. 1983), as reprinted in Vector Quantization (IEEE Press, 1990), 1990, pp. 233 Trowbridge, David, “Using Andrew for Development of Educational 236. Applications”. Center for Design of Educational Computing, Wu et al., “Automatic Generation of Synthesis Units and Prosodic Carnegie-Mellon University (CMU-ITC-85-065), Jun. 2, 1985, pp. Information for Chinese Concatenative Synthesis'. Speech Commu 1-6. nication, vol. 35. No. 3-4, Oct. 2001, pp. 219-237. Tsao et al., “Matrix Quantizer Design for LPC Speech Using the Ahlstrom et al., “Overcoming Touchscreen User Fatigue by Work Generalized Lloyd Algorithm'. (IEEE Transactions on Acoustics, place Design”, CHI '92 Posters and Short Talks of the 1992 SIGCHI Speech and Signal Processing, Jun. 1985), as reprinted in Vector Conference on Human Factors in Computing Systems, 1992, pp. Quantization (IEEE Press, 1990), 1990, pp. 237-245. 101-102. Turletti. Thierry, “The INRIA Videoconferencing System (IVS)”. Yang et al., “Auditory Representations of Acoustic Signals'. IEEE Oct. 1994, pp. 1-7. Transactions of Information Theory, vol. 38, No. 2, Mar. 1992, pp. Uslanet al., “A Review of Henter-Joyce's MAGic for Windows NT". 824-839. Journal of Visual Impairment and Blindness, Dec. 1999, pp. 666-668. Yang et al., “Hidden Markov Model for Mandarin Lexical Tone Uslanet al., “A Review of Supernova Screen Magnification Program Recognition', IEEE Transactions on Acoustics, Speech and Signal for Windows'. Journal of Visual Impairment & Blindness, Feb. 1999, Processing, vol. 36, No. 7, Jul. 1988, pp.988-992. pp. 108-110. Yiourgalis et al., “Text-to-Speech system for Greek”, ICASSP 91. Uslan et al., “A Review of Two Screen Magnification Programs for vol. 1, May 14-17, 1991, pp. 525-528. Windows 95: Magnum 95 and LP-Windows'. Journal of Visual Ahmed et al., “Intelligent Natural Language Query Processor'. Impairment & Blindness, Sep.-Oct. 1997, pp. 9-13. TENCON 89, Fourth IEEE Region 10 International Conference, Veiga, Alex, "AT&T Wireless Launching Music Service', available Nov. 22-24, 1989, pp. 47-49. at , Jan. 3, 2012, 3 pages. I, Apr. 28-May 3, 2007, pp. 657-666. Zelig, “A Review of the Palm Treo 750 v', available at , Feb. 5, 2007, 3 pages.

(56) References Cited Bederson et al., “Pad++: A Zooming Graphical Interface for Explor ing Alternate Interface Physics”. UIST 94 Proceedings of the 7th OTHER PUBLICATIONS Annual ACM Symposium on User Interface Software and Technol ogy, Nov. 1994, pp. 17-26. ANHUIUSTC IFL YTEK Co. Ltd., “Flytek Research Center Infor Bederson et al., “The Craft of Information Visualization'. Elsevier mation Datasheet', available at , retrieved on Oct. 15, 2004, 3 pages. Apple, “Voice0ver', available at (http://www.apple.com/accessibil Extended European Search Report (includes European Search ity/voiceover/>, Feb. 2009, 5 pages. Report and European Search Opinion) received for European Patent Apple Computer, Inc., "Apple iPod Technical Specifications, Application No. 121861 13.2, mailed on Apr. 28, 2014, 14 pages. iPod 20GB and 60GB Mac + PC, available at , 2005, 3 pages. Report and European Search Opinion) received for European Patent Benelet al., “OptimalSize and Spacing of Touchscreen Input Areas'. Application No. 13155688.8, mailed on Aug. 22, 2013, 11 pages. Human-Computer Interaction—INTERACT, 1987, pp. 581-585. “Windows XP. A Big Surprise!—Experiencing Amazement from Beringer et al., “Operator Behavioral Biases. Using High-Resolution Touch Input Devices'. Proceedings of the Human Factors and Ergo Windows XP, New Computer, No. 2, Feb. 28, 2002, 8 pages. nomics Society 33rd Annual Meeting, 1989, 3 pages. Schnelle, Dirk, “Context Aware Voice User Interfaces for Workflow Beringer, Dennis B., “Target Size, Location, Sampling Point and Support’. Dissertation paper, Aug. 27, 2007, 254 pages. Instruction Set: More Effects on Touch Panel Operation'. Proceed “Minimum Phase'. Wikipedia the free Encyclopedia, Last Modified ings of the Human Factors and Ergonomics Society 34th Annual on Jan. 12, 2010, and retrieved on Jul. 28, 2010, available online at Meeting, 1990, 5 pages. , 2 pages. Bernstein, Macrophone, “Speech Corpus', IEEE/ICASSP, Apr. 22. Non-Final Office Action received for U.S. Appl. No. 13/492,809, 1994, pp. 1-81 to 1-84. mailed on Nov. 27, 2013, 8 pages. Berry et al., “Symantec”. New version of MORE.TM, Apr. 10, 1990, Final Office Action received for U.S. Appl. No. 13/725,481, mailed 1 page. on Dec. 19, 2013, 16 pages. Best Buy, “When it Comes to Selecting a Projection TV, Toshiba Apple Computer, Inc., "iTunes 2: Specification Sheet”, 2001, 2 Makes Everything Perfectly Clear'. Previews of New Releases, pageS. available at , 2005, pp. 1-32. face'. Computer Graphics (SIGGRAPH '93 Proceedings), vol. 27. Apple Computer, Inc., "Apple Announces iTunes 2'. Press Release, 1993, pp. 73-80. Oct. 23, 2001, 2 pages. Birrell, Andrew, “Personal Jukebox (PJB)', available at , retrieved on TTS", 5th ISCA Speech Synthesis Workshop, Pittsburgh, PA, Jun. Apr. 29, 2006, 4 pages. 14-16, 2004, 2 pages. Bocchieri et al., “Use of Geographical Meta-Data in ASR Language Baechtleet al., “Adjustable Audio Indicator'. IBMTechnical Disclo and Acoustic Models', IEEE International Conference on Acoustics sure Bulletin, Jul. 1, 1984, 2 pages. Speech and Signal Processing, 2010, pp. 51 18-5121. Baeza-Yates, Ricardo, “Visualization of Large Answers in Text Data Bociurkiw, Michael, “Product Guide: Vanessa Matz’, available at bases'. AVI '96 Proceedings of the Workshop on Advanced Visual . Interfaces, 1996, pp. 101-107. retrieved on Jan. 23, 2003, 2 pages. Bahl et al., “Recognition of a Continuously Read Natural Corpus', Borden IV, G.R., “An Aural User Interface for Ubiquitous Comput IEEE International Conference on Acoustics, Speech, and Signal ing”. Proceedings of the 6th International Symposium on Wearable Processing, vol. 3, Apr. 1978, pp. 422-424. Computers, IEEE, 2002, 2 pages. Bajarin, Tim, “With Low End Launched, Apple Turns to Portable Borenstein, Nathaniel S.“Cooperative Work in the Andrew Message Future”, PC Week, vol. 7, Oct. 1990, p. 153 (1). System”, Information Technology Center and Computer Science Barthel, B., “Information Access for Visually Impaired Persons: Do Department, Carnegie Mellon University; Thyberg, Chris A. Aca We Still Keep a “Document” in "Documentation”?”. Professional demic Computing, Carnegie Mellon University, 1988, pp. 306–323. Communication Conference, Sep. 1995, pp. 62-66. Boy, Guy A., “Intelligent Assistant Systems”. Harcourt Brace Baudelet al., “2 Techniques for Improved HC Interaction: Toolglass Jovanovicy, 1991, 1 page. & Magic Lenses: The See-Through Interface”. Apple Inc., Video Apple Computer, Inc., "Apple Introduces iTunes World's Best and Clip, CHI’94 Video Program on a CD, 1994. Easiest to Use Jukebox Software', Macworld Expo, Jan. 9, 2001, 2 Beck et al., “Integrating Natural Language, Query Processing, and pageS. Semantic Data Models”, COMCON Spring 90. IEEE Computer Brown et al., “Browing Graphs. Using a Fisheye View”. Apple Inc., Society International Conference, 1990, Feb. 26-Mar. 2, 1990, pp. Video Clip, Systems Research Center, CHI '92 Continued Proceed 538-543. ings on a CD, 1992. US 9,318.108 B2 Page 40

(56) References Cited Compaq Inspiration Technology, "Personal Jukebox (PJB)—Sys tems Research Center and PAAD”, Oct. 13, 2000, 25 pages. OTHER PUBLICATIONS Conkie et al., “Preselection of Candidate Units in a Unit Selection Based Text-to-Speech Synthesis System”. ISCA, 2000, 4 pages. Brown et al., “Browshing Graphs. Using a Fisheye View'. CHI '93 Conklin, Jeffrey, "A Survey of Hypertext”. MCC Software Technol Proceedings of the INTERACT 93 and CHI '93 Conference on ogy Program, Dec. 1987, 40 pages. Human Factors in Computing Systems, 1993, p. 516. Copperiet al., "CELP Coding for High Quality Speech at 8 kbits/s'. Burger, D., “Improved Access to Computers for the Visually Handi Proceedings of IEEE International Acoustics, Speech and Signal capped: New Prospects and Principles', IEEE Transactions on Reha Processing Conference, Apr. 1986), as reprinted in Vector Quantiza bilitation Engineering, vol. 2, No. 3, Sep. 1994, pp. 111-118. tion (IEEE Press), 1990, pp. 324-327. Busemann et al., “Natural Language Diaglogue Service for Appoint Corr, Paul. “Macintosh Utilities for Special Needs Users', available ment Scheduling Agents”, Technical Report RR-97-02, Deutsches at (http://homepage.mac.com/corrp/macSupt/columns specneeds. Forschungszentrum fur Kunstliche Intelligenz GmbH. 1997.8 pages. html>, Feb. 1994 (content updated Sep. 19, 1999), 4 pages. Butler, Travis, “Archos Jukebox 6000 Challenges Nomad Jukebox”. Creative, “Creative NOMAD MuVo, available at , Aug. 13, 2001, 5 archive.org/web/20041024075901/www.creative.com/products/ pageS. product.asp?category=213&subcategory=216&product=4983>. Butler, Travis, “Portable MP3: The Nomad Jukebox', available at retrieved on Jun. 7, 2006, 1 page. , retrieved on Jun. 6, 2006, 1 page. ference on Multi-User Interfaces and Applications, 1990, pp. 11-34. Creative, “Digital MP3 Player', available at . Jun. 1999, 40 pages. 1990, pp. 328-330. Creative Technology Ltd., “Creative NOMADR II: Getting Card et al., “Readings in Information Visualization Using Vision to Started User Guide (on Line Version)', available at , Apr. 2000, 46 pages. Presentation of Visual Information'. UIST '95 Proceedings of the 8th Creative Technology Ltd., “Nomad Jukebox”. User Guide, Version Annual ACM Symposium on User Interface and Software Technol 1.0, Aug. 2000, 52 pages. ogy, Nov. 14-17, 1995, pp. 217-226. Croft et al., “Task Support in an Office System'. Proceedings of the Carpendale et al., “Extending Distortion Viewing from 2D to 3D”. Second ACM-SIGOA Conference on Office Information Systems, IEEE Computer Graphics and Applications, Jul/Aug. 1997, pp. 1984, pp. 22-24. 42-51. Crowley et al., “MMConf: An Infrastructure for Building Shared Carpendale et al., “Making Distortions Comprehensible'. IEEE Pro Multimedia Applications', CSCW 90 Proceedings, Oct. 1990, pp. ceedings of Symposium on Visual Languages, 1997. 10 pages. 329-342. Casner et al., “N-Way Conferencing with Packet Video'. The Third Cuperman et al., “Vector Predictive Coding of Speech at 16 kbits/s'. International Workshop on Packet Video, Mar. 22-23, 1990, pp. 1-6. (IEEE Transactions on Communications, Jul. 1985), as reprinted in Chakarova et al., “Digital Still Cameras—Downloading Images to a Vector Quantization (IEEE Press, 1990), 1990, pp. 300-311. Computer'. Multimedia Reporting and Convergence, available at Abut et al., “Low-Rate Speech Encoding Using Vector Quantization , retrieved on May 9, 2005, 2 pages. Acoustics, Speech and Signal Processing Conference, Apr. 1986), as Apple Computer, Inc., "Apple's iPod Available in Stores Tomorrow”, reprinted in Vector Quantization IEEE Press, 1990, pp. 312-315. Press Release, Nov. 9, 2001, 1 page. Davis et al., “Stone Soup Translation'. Department of Linguistics, Chartier, David, “Using Multi-Network Meebo Chat Service on Your Ohio State University, 2001, 11 pages. iPhone', available at , Jul. 4, 2007. able at , Oct. 5 pages. 13, 2000, 8 pages. Apple Computer, Inc., “Inside Macintosh'', vol. VI, 1985. Degani et al., “Soft Controls for Hard Displays: Still a Challenge”. Apple Computer, Inc., "iTunes 2, Playlist Related Help Screens', Proceedings of the 36th Annual Meeting of the Human Factors Soci iTunes v2.0, 2000-2001, 8 pages. ety, 1992, pp. 52-56. ABCOMPTY, Ltd.. “12.1"925 Candela MobilePC, LCDHardware. Del Strother, Jonathan, "Coverflow”, available at , 1998, 28 pages. on Feb. 11, 2004. 1 page. Dickinson et al., “Palmtips: Tiny Containers for All Your Data”, PC Cisco Systems, Inc., "Cisco Unity Unified Messaging User Guide”. Magazine, vol. 9, Mar. 1990, p. 218(3). Release 4.0(5), Apr. 14, 2005, 152 pages. Digital Equipment Corporation, “OpenVMS RTL DECtalk (DTKS) Cisco Systems, Inc., “Installation Guide for Cisco Unity Unified Manual', May 1993, 56 pages. Messaging with Microsoft Exchange 2003/2000 (With Failover Con Donahue et al., “Whiteboards: A Graphical Database Tool”, ACM figured)'. Release 4.0(5), Apr. 14, 2005, 152 pages. Transactions on Office Information Systems, vol. 4. No. 1, Jan. 1986, Cisco Systems, Inc., “Operations Manager Tutorial, Cisco's IPC pp. 24-41. Management Solution', 2006, 256 pages. Dourish et al., “Portholes: Supporting Awareness in a Distributed Coleman, David W., “Meridian Mail Voice Mail System Integrates Work Group’, CHI 1992. May 1992, pp. 541-547. Voice Processing and Personal Computing'. Speech Technology, vol. Dusan et al., “Multimodal Interaction on PDA's Integrating Speech 4. No. 2, Mar/Apr. 1988, pp. 84-87. and Pen Inputs'. Eurospeech Geneva, 2003, 4 pages. Compaq, “Personal Jukebox', available at (http://research.compaq. Dyslexic.com, "AlphaSmart 3000 with CoWriter SmartApplet: Don com/SRC/pb/>, 2001, 3 pages. Johnston Special Needs', available at

(56) References Cited Gmail, "About Group Chat”, available at , retrieved on Dec. 19, 2002, 1 page. neering and Comp. Sciences, University of California, Berkley, Nov. Engst, Adam C. "SoundJam Keeps on Jammin', available at , Jun. 19, 2000, 3 pages. Hall, William S. "Adapt Your Program for Worldwide Use with Ericsson Inc., "Cellular Phone with Integrated MP3 Player', Windows.TM. Internationalization Support'. Microsoft Systems Research Disclosure Journal No. 41815, Feb. 1999, 2 pages. Journal, vol. 6, No. 6, Nov./Dec. 1991, pp. 29-58. Eslambolchilar et al., “Making Sense of Fisheye Views”. Second Haoui et al., “Embedded Coding of Speech: A Vector Quantization Dynamics and Interaction Workshop at University of Glasgow, Aug. Approach'. (Proceedings of the IEEE International Acoustics, 2005, 6 pages. Speech and Signal Processing Conference, Mar. 1985), as reprinted Eslambolchilar et al., “Multimodal Feedback for Tilt Controlled in Vector Quantization (IEEE Press, 1990), 1990, pp. 297-299. Speed Dependent Automatic Zooming, UIST'04, Oct. 24-27, 2004, Hartson et al., “Advances in Human-Computer Interaction'. Chap 2 pages. ters 1, 5, and 6, vol. 3, 1992, 121 pages. Fanty et al., “A Comparison of DFT. PLP and Cochleagram for Heger et al., “KNOWBOT: An Adaptive Data Base Interface'. Alphabet Recognition', IEEE, Nov. 1991. Nuclear Science and Engineering, V. 107, No. 2, Feb. 1991, pp. Findlater et al., “Beyond QWERTY: Augmenting Touch-Screen with 142-157. Keyboards with Multi-Touch Gestures for Non-Alphanumeric Hendrix et al., “The Intelligent Assistant: Technical Considerations Input'. CHI 12, Austin, Texas, USA, May 5-10, 2012, 4 pages. Involved in Designing Q&A's Natural-Language Interface'. Byte Fisher et al., “Virtual Environment Display System”, Interactive 3D Magazine, Issue 14, Dec. 1987, 1 page. Graphics, Oct. 23-24, 1986, pp. 77-87. Heyer et al., “Exploring Expression Data: Identification and Analysis Forsdick, Harry, “Explorations into Real-Time Multimedia of Coexpressed Genes', Genome Research, vol. 9, 1999, pp. 1106 Conferencing'. Proceedings of the Ifip Tc 6 International Sympo 1115. sium on Computer Message Systems, 1986, 331 pages. Hill, R. D., "Some Important Features and Issues in User Interface Furnas et al., “Space-Scale Diagrams: Understanding Multiscale Management System'. Dynamic Graphics Project, University of Interfaces”, CHI '95 Proceedings of the SIGCHI Conference on Toronto, CSRI, vol. 21, No. 2, Apr. 1987, pp. 116-120. Human Factors in Computing Systems, 1995, pp. 234-241. Hinckley et al., “A Survey of Design Issues in Spatial Input', UIST Furnas, George W. “Effective View Navigation'. Proceedings of the '94 Proceedings of the 7th Annual ACM Symposium on User Inter ACM SIGCHI Conference on Human Factors in Computing Sys face Software and Technology, 1994, pp. 213-222. tems, Mar. 1997, pp. 367-374. Hiroshi, “TeamWork Station: Towards a Seamless Shared Furnas, George W. “Generalized Fisheye Views”. CHI '86 Proceed Workspace”, NTT Human Interface Laboratories, CSCW 90 Pro ings of the SIGCHI Conference on Human Factors in Computing ceedings, Oct. 1990, pp. 13-26. Systems, vol. 17, No. 4, Apr. 1986, pp. 16-23. Holmes, “Speech System and Research”, 1955, pp. 129-135, 152 Furnas, George W. “The Fisheye Calendar System”. Bellcore Tech 153. nical Memorandum, Nov. 19, 1991. Hon et al., “Towards Large Vocabulary Mandarin Chinese Speech Gardner, Jr., P. C., “A System for the Automated Office Environ Recognition'. Conference on Acoustics, Speech, and Signal Process ment', IBM Systems Journal, vol. 20, No. 3, 1981, pp. 321-345. ing, ICASSP-94, IEEE International, vol. 1, Apr. 1994, pp. 545-548. Garretson, R., “IBM Adds 'Drawing Assistant Design Tool to Hopper, Andy, "Pandora—An Experimental System for Multimedia Graphic Series'. PC Week, vol. 2, No. 32, Aug. 13, 1985, 1 page. Applications'. Olivetti Research Laboratory, Apr. 1990, pp. 19-34. Gaver et al., “One Is Not Enough: Multiple Views in a Media Space'. Howard, John H., "(Abstract) An Overview of the Andrew File Sys INTERCHI, Apr. 24-29, 1993, pp. 335-341. tem”, Information Technology Center, Carnegie Mellon University; Gaver et al., “Realizing a Video Environment: EuroPARC's RAVE (CMU-ITC-88-062) to Appear in a future issue of the ACM Trans System”. Rank Xerox Cambridge EuroPARC, 1992, pp. 27-35. actions on Computer Systems, 1988, pp. 1-6. Giachin et al., “Word Juncture Modeling Using Inter-Word Context Huang et al., “Real-Time Software-Based Video Coder for Multime Dependent Phone-Like Units”. Cselt Technical Reports, vol. 20, No. dia Communication Systems'. Department of Computer Science and 1, Mar. 1992, pp. 43-47. Information Engineering, 1993, 10 pages. Gillespie, Kelly, “Adventures in Integration', Data Based Advisor, Hukin, R. W. “Testing an Auditory Model by Resynthesis”, Euro vol. 9, No. 9, Sep.1991, pp. 90-92. pean Conference on Speech Communication and Technology, Sep. Gillespie, Kelly, “InternationalizeYour Applications with Unicode”. 26-29, 1989, pp. 243-246. Data Based Advisor, vol. 10, No. 10, Oct. 1992, pp. 136-137. Hunt, “Unit Selection in a Concatenative Speech Synthesis System Gilloire et al., “Innovative Speech Processing for Mobile Terminals: Using a Large Speech Database'. Copyright 1996 IEEE. “To appear An Annotated Bibliography”. Signal Processing, vol. 80, No. 7, Jul. in Proc. ICASSP-96, May 7-10, Atlanta, GA” ATR Interpreting Tele 2000, pp. 1149-1166. communications Research Labs, Kyoto Japan, 1996, pp. 373-376. Glinert-Stevens, Susan, “Microsoft Publisher: Desktop Wizardry'. IBM, “Why Buy. ThinkPad', available at , retrieved on Dec. 19, 2002, 2 pages. US 9,318.108 B2 Page 42

(56) References Cited International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2007/026243, mailed on Mar. 31, OTHER PUBLICATIONS 2008, 10 pages. International Search Report and Written Opinion received for PCT IBM Corporation, "Simon Says Here's How”. Users Manual, 1994, Patent Application No. PCT/US2007/077424, mailed on Jun. 19, 3 pages. 2008, 13 pages. iChat AV. "Video Conferencing for the Rest of Us', Apple Mac OS Invitation to Pay Additional Fees received for PCT Application No. X—iChat AV, available at , retrieved on Dec. 25, 2007, 5 pages. International Search Report and Written Opinion received for PCT iPhonechat, “iChat for iPhone in JavaScript', available at , Mar. 17, 2007, 6 pages. Patent Application No. PCT/US2008/000045, mailed Jun. 12, 2008, Palay et al., “The Andrew Toolkit: An Overview”. Information Tech 7 pages. nology Center, Carnegie-Mellon University, 1988, pp. 1-15. International Search Report and Written Opinion received for PCT Palm, Inc., “User Guide:Your PalmR) Treo.TM. 755p Smartphone”, Patent Application No. PCT/US2008/000047, mailed on Sep. 11, 2005-2007, 304 pages. 2008, 12 pages. Panasonic, "Toughbook 28: Powerful, Rugged and Wireless'. Invitation to Pay Additional Fees received for PCT Patent Applica Panasonic: Toughbook Models, available at , retrieved on Dec. 19. International Search Report and Written Opinion received for PCT 2002, 3 pages. Patent Application No. PCT/US2008/000059, mailed on Sep. 19, Parks et al., “Classification of Whale and Ice Sounds with a cochlear 2008, 18 pages. Model”, IEEE, Mar. 1992. International Search Report and Written Opinion received for PCT Patterson et al., “Rendezvous: An Architecture for Synchronous Patent Application No. PCT/US2008/000061, mailed on Jul. 1, 2008, Multi-User Applications”, CSCW '90 Proceedings, 1990, pp. 317 13 pages. 328. International Search Report and Written Opinion received for PCT International Search Report received for PCT Patent Application No. Patent Application No. PCT/US2008/050083, mailed on Jul. 4, 2008, PCT/US2002/033330, mailed on Feb. 4, 2003, 6 pages. 9 pages. International Search Report and Written Opinion received for PCT International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2004/016519, mailed on Nov. 3, Patent Application No. PCT/US2011/020350, mailed on Jun. 30, 2005, 16 pages. 2011, 17 pages. Partial International Search Report and Invitation to Pay Additional Invitation to Pay Additional Fees and Partial International Search Fees received for PCT Patent Application No. PCT/US2004/016519, Report received for PCT Patent Application No. PCT/US2011/ mailed on Aug. 4, 2005, 6 pages. 020350, mailed on Apr. 14, 2011, 5 pages. International Search Report and Written Opinion received for PCT International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2005/038819, mailed on Apr. 5, Patent Application No. PCT/US2011/020861, mailed on Aug. 2, 2006, 12 pages. 2012, 11 pages. International Search Report received for PCT Patent Application No. International Search Report received for PCT Patent Application No. PCT/US2005/046797, mailed on Nov. 24, 2006, 6 pages. PCT/US2011/037014, mailed on Oct. 4, 2011, 16 pages. Invitation to Pay Additional Fees and Partial Search Report received Invitation to Pay Additional Search Fees received for PCT Applica for PCT Application No. PCT/US2005/046797, mailed on Jul. 3, tion No. PCT/US2011/037014, mailed on Aug. 2, 2011, 6 pages. 2006, 6 pages. Extended European Search Report (includes European Search Written Opinion received for PCT Patent Application No. PCT/ Report and European Search Opinion) received for European Patent US2005/046797, mailed on Nov. 24, 2006, 9 pages. Application No. 06256215.2, mailed on Feb. 20, 2007, 6 pages. International Search Report and Written Opinion received for PCT Extended European Search Report (includes Supplementary Euro Patent Application No. PCT/US2006/048669, mailed on Jul. 2, 2007, pean Search Report and Search Opinion) received for European 12 pages. Patent Application No. 07863218.9, mailed on Dec. 9, 2010, 7 pages. International Search Report and Written Opinion received for PCT International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2006/048670, mailed on May 21, Patent Application No. PCT/US2012/034028, mailed on Jun. 11, 2007, 11 pages. 2012, 9 pages. Invitation to Pay Addition Fees and Partial International Search International Search Report and Written Opinion received for PCT Report received for PCT Patent Application No. PCT/US2006/ Patent Application No. PCT/US2012/04.0931, mailed on Feb. 1, 048738, mailed on Jul. 10, 2007, 4 pages. 2013, 4 pages (International Search Report only). International Search Report and Written Opinion received for PCT International Search Report and Written Opinion received for PCT Patent Application No. PCT/US2006/048753, mailed on Jun. 19, Patent Application No. PCT/US2012/043098, mailed on Nov. 14, 2007, 15 pages. 2012, 9 pages. US 9,318.108 B2 Page 43

(56) References Cited Seagrave, Jim, “A Faster Way to Search Text', EXE, vol. 5, No. 3, Aug. 1990, pp. 50-52. OTHER PUBLICATIONS Sears et al., “High Precision Touchscreens: Design Strategies and Comparisons with a Mouse'. International Journal of Man-Machine Ahlberg et al., “The Alphaslider: A Compact Rapid Selector', CHI Studies, vol. 34, No. 4, Apr. 1991, pp. 593-613. '94 Proceedings of the SIGCHI Conference Factors in Computing Sears et al., “Investigating Touchscreen Typing: The Effect of Key Systems, Apr. 1994, pp. 365-371. board Size on Typing Speed”. Behavior & Information Technology, International Search Report and Written Opinion received for PCT vol. 12, No. 1, 1993, pp. 17-22. Patent Application No. PCT/US2013/041225, mailed on Aug. 23. Sears et al., “Touchscreen Keyboards'. Apple Inc., Video Clip, 2013, 3 pages (International Search Report only). Human-Computer Interaction Laborato, on a CD, Apr. 1991. Invitation to Pay Additional Fees received for PCT Patent Applica Seide et al., “Improving Speech Understanding by Incorporating tion No. PCT/US2013/047659, mailed on Feb. 27, 2014, 7 pages. Invitation to Pay Additional Fees received for PCT Application No. Database Constraints and Dialogue History”. Proceedings of Fourth PCT/US2013/052558, mailed on Nov. 7, 2013, 6 pages. International Conference on Philadelphia, 1996, pp. 1017-1020. Scheifler, R. W. “The X Window System', MIT Laboratory for Shiraki et al., “LPC Speech Coding Based on Variable-Length Seg Computer Science and Gettys, Jim Digital Equipment Corporation ment Quantization'. (IEEE Transactions on Acoustics, Speech and and MIT Project Athena; ACM Transactions on Graphics, vol. 5, No. Signal Processing, Sep. 1988), as reprinted in Vector Quantization 2, Apr. 1986, pp. 79-109. (IEEE Press, 1990), 1990, pp. 250-257. Schluter et al., “Using Phase Spectrum Information for Improved Shneiderman, Ben, “Designing the User Interface: Strategies for Speech Recognition Performance', IEEE International Conference Effective Human-Computer Interaction'. Second Edition, 1992. 599 on Acoustics, Speech, and Signal Processing, 2001, pp. 133-136. pageS. Schmandt et al., “A Conversational Telephone Messaging System'. Shneiderman, Ben, “Designing the User Interface: Strategies for IEEE Transactions on Consumer Electronics, vol. CE-30, Aug. 1984, Effective Human-Computer Interaction'. Third Edition, 1998, 669 pp. XXi-XXiv. pageS. Schmandt et al., “Phone Slave: A Graphical Telecommunications Shneiderman, Ben, “Direct Manipulation for Comprehensible, Pre Interface”. Society for Information Display, International Sympo dictable and Controllable User Interfaces'. Proceedings of the 2nd sium Digest of Technical Papers, Jun. 1984, 4 pages. International Conference on Intelligent User Interfaces, 1997, pp. Schmandt et al., “Phone Slave: A Graphical Telecommunications 33-39. Interface'. Proceedings of the SID, vol. 26, No. 1, 1985, pp. 79-82. Shneiderman, Ben, “Sparks of Innovation in Human-Computer Inter Schmid, H., “Part-of-speech tagging with neural networks', COL action', 1993, (Table of Contents, Title Page, Ch. 4, Ch. 6 and List of ING 94 Proceedings of the 15th conference on Computational lin References). guistics—vol. 1, 1994, pp. 172-176. Shneiderman, Ben, “The Eyes Have It: A Task by Data Type Tax Schooler et al., “A Packet-switched Multimedia Conferencing Sys onomy for Information Visualizations”, IEEE Proceedings of Sym tem”. By Eve Schooler, et al; ACM SIGOIS Bulletin, vol. I, No. 1, posium on Visual Languages, 1996, pp. 336-343. Jan. 1989, pp. 12-22. Shneiderman, Ben, “Touch Screens Now Offer Compelling Uses'. Schooler et al., “An Architecture for Multimedia Connection Man IEEE Software, Mar. 1991, pp. 93-94. agement”. Proceedings IEEE 4th ComSoc. International Workshop on Shoham et al., “Efficient Bit and Allocation for an Arbitrary Set of Multimedia Communications, Apr. 1992, pp. 271-274. Quantizers'. (IEEE Transactions on Acoustics, Speech, and Signal Schooler et al., “Multimedia Conferencing: Has it Come of Age?”, Processing, Sep. 1988) as reprinted in Vector Quantization (IEEE Proceedings 24th Hawaii International Conference on System Sci Press, 1990), 1990, pp. 289-296. ences, vol. 3, Jan. 1991, pp. 707-716. Simkovitz, Daniel, “LP-DOS Magnifies the PC Screen”, IEEE, 1992, Schooler et al., “The Connection Control Protocol: Architecture pp. 203-204. Overview”, USC/Information Sciences Institute, Jan. 28, 1992, pp. Singh et al., “Automatic Generation of Phone Sets and Lexical Tran 1-6. scriptions'. Acoustics, Speech and Signal Processing (ICASSPO0), Schooler, Eve, "A Distributed Architecture for Multimedia Confer 2000, 1 page. ence Control”. ISI Research Report, Nov. 1991, pp. 1-18. Sinitsyn, Alexander, "A Synchronization Framework for Personal Schooler, Eve M.. “Case Study: Multimedia Conference Control in a Mobile Servers'. Proceedings of the Second IEEE Annual Confer Packet-Switched Teleconferencing System”, Journal of ence on Pervasive Computing and Communications Workshops, Internetworking: Research and Experience, vol. 4. No. 2, Jun. 1993, Piscataway, 2004, pp. 1, 3 and 5. pp. 99-120. Slaney et al., “On the Importance of Time—A Temporal Represen Schooler, Eve M.. “The Impact of Scaling on a Multimedia Connec tation of Sound'. Visual Representation of Speech Signals, 1993, pp. tion Architecture'. Multimedia Systems, vol. 1, No. 1, 1993, pp. 2-9. 95-116. Schütze, H., “Distributional part-of-speech tagging”. EACL 95 Pro Smeaton, Alan F. “Natural Language Processing and Information ceedings of the seventh conference on European chapter of the Asso Retrieval”. Information Processing and Management, vol. 26, No. 1, ciation for Computational Linguistics, 1995, pp. 141-148. 1990, pp. 19-20. Schütze, Hinrich, “Part-of-speech induction from scratch'. ACL 93 Smith et al., “Guidelines for Designing User Interface Software'. Proceedings of the 31st annual meeting on Association for Compu User Lab, Inc., Aug. 1986, pp. 1-384. tational Linguistics, 1993, pp. 251-258. Smith et al., “Relating Distortion to Performance in Distortion Ori Schwartz et al., "Context-Dependent Modeling for Acoustic-Pho ented Displays”. Proceedings of Sixth Australian Conference on netic Recognition of Continuous Speech'. IEEE International Con Computer-Human Interaction, Nov. 1996, pp. 6-11. ference on Acoustics, Speech, and Signal Processing, vol. 10, Apr. Sony Eiicsson Corporate, “Sony Ericsson to introduce Auto pairing. 1985, pp. 1205-1208. TM. to Improve Bluetooth.TM. Connectivity Between Headsets and Schwartz et al., “Improved Hidden Markov Modeling of Phonemes Phones'. Press Release, available at . ence on Acoustics, Speech, and Signal Processing, vol. 9, 1984, pp. Sep. 28, 2005, 2 pages. 21-24. Soong et al., “A High Quality Subband Speech Coder with Backward Schwartz et al., “The N-Best Algorithm: An Efficient and Exact Adaptive Predictor and Optimal Time-Frequency Bit Assignment”. Procedure For Finding The N Most Likely Sentence Hypotheses'. (Proceedings of the IEEE International Acoustics, Speech, and Sig IEEE, 1990, pp. 81-84. nal Processing Conference, Apr. 1986), as reprinted in Vector Quan Scott et al., “Designing Touch Screen Numeric Keypads: Effects of tization (IEEE Press, 1990), 1990, pp. 316-319. Finger Size, Key Size, and KeySpacing”. Proceedings of the Human Spiller, Karen, “Low-Decibel Earbuds Keep Noise at a Reasonable Factors and Ergonomics Society 41st Annual Meeting, Oct. 1997, pp. Level”, available at shttp://www.nashuatelegraph.com/apps/pbcs. 360-364. dll/article?Date=20060813&Cate.>, Aug. 13, 2006, 3 pages. US 9,318.108 B2 Page 44

(56) References Cited Kirstein et al., “Piloting of Multimedia Integrated Communications for European Researchers”. Proc. INET '93, 1993, pp. 1-12. OTHER PUBLICATIONS Kjelldahl et al., “Multimedia Principles, Systems, and Applica tions'. Proceedings of the 1991 Eurographics Workshop on Multi Ahlberg et al., “Visual Information Seeking: Tight Coupling of media Systems, Applications, and Interaction, Apr. 1991. Dynamic Query Filters with Starfield Displays'. Proceedings of the Kline et al., “Improving GUI Accessibility for People with Low SIGCHI Conference on Human Factors in Computing Systems, Apr. Vision”, CHI '95 Proceedings of the SIGCHI Conference on Human 24-28, 1994, pp. 313-317. Factors in Computing Systems, May 7-11, 1995, pp. 114-121. Srinivas et al., “Monet: A Multi-Media System for Conferencing and Kline et al., “UnWindows 1.0: X Windows Tools for Low Vision Application Sharing in Distributed Systems”. CERC Technical Users'. ACM SIGCAPH Computers and the Physically Handi Report Series Research Note, Feb. 1992. Stealth Computer Corporation, “Peripherals for Industrial Keyboards capped, No. 49, Mar. 1994, pp. 1-5. & Pointing Devices', available at , retrieved on Dec. 19, 2002, 6 pages. Intelligence, 2nd ed., McGraw-Hill, Inc., 1983-1991. Steinberg, Gene, “Sonicblue Rio Car (10 GB, Reviewed: 6 GB), Kroon et al., “Quantization Procedures for the Excitation in CELP available at , Dec. 12, 2000, 2 pages. Signal Processing Conference, Apr. 1987), as reprinted in Vector Stent et al., “Geo-Centric LanguageModels for Local Business Voice Quantization (IEEE Press, 1990), 1990, pp. 320-323. Search”, AT&T Labs Research, 2009, pp. 389-396. Kuo et al., “A Radical-Partitioned code Block Adaptive Neural Net Stone et al., “The Movable Filter as a User Interface Tool, CHI '94 work Structure for Large-Volume Chinese Character Recognition'. Human Factors in Computing Systems, 1994, pp. 306-312. International Joint Conference on Neural Networks, vol. 3, Jun. Su et al., “A Review of ZoomText Xtra Screen Magnification Pro 1992, pp. 597-601. gram for Windows 95”, Journal of Visual Impairment & Blindness, Kuo et al., “A Radical-Partitioned Neural Network System Using a Feb. 1998, pp. 116-119. Modified Sigmoid Function and a Weight-Dotted Radical Selector Su, Joseph C., “A Review of Telesensory's Vista PCI Screen Magni for Large-Volume Chinese Character Recognition VLSI", IEEE Int. fication System”, Journal of Visual Impairment & Blindness, Oct. Symp. Circuits and Systems, Jun. 1994, pp. 3862-3865. 1998, pp. 705, 707-710. Kurlander et al., “Comic Chat”. Online, 1996 Retrieved on: Feb. 4, JABRA, “Bluetooth Introduction', 2004, 15 pages. 2013, SIGGRAPH '96 Proceedings of the 23rd annual conference JABRA Corporation, “FreeSpeak: BT200 User Manual”, 2002, 42 on Computer graphics and interactive techniques, Retrieved from: pageS. http://delivery.acm.org/10. 1145/240000/237260, p225-kurlander. Jaybird, “Everything Wrong with AIM. Because We've All Thought pdf), 1996, pp. 225-236. About It”, available at , May 24, 2006, 3 pages. lary Recognition'. International Conference on Acoustics, Speech, Jeffay et al., "Kernel Support for Live Digital Audio and Video'. In and Signal Processing, vol. 1, May 1995, pp. 560-563. Proc. of the Second Intl. Workshop on Network and Operating Sys Lafferty et al., “Conditional Random Fields: Probabilistic Models for tem. Support for Digital Audio and Video, vol. 614, Nov. 1991, pp. Segmenting and Labeling Sequence Data'. Proceedings of the 18th 10-21. International Conference on Machine Learning, 2001, 9 pages. Jelineket al., “Interpolated Estimation of Markov Source Parameters Lamel et al., “Generation and synthesis of Broadcast Messages'. from Sparse Data'. In Proceedings of the Workshop on Pattern Rec Proceedings of ESCA-NATO Workshop: Applications of Speech ognition in Practice May 1980, pp. 381-397. Technology, Sep. 1, 1993, 4 pages. Johnson, Jeff A., “A Comparison of User Interfaces for Panning on a Lamping et al., "Laying Out and Visualizing Large Trees Using a Touch-Controlled Display”, CHI '95 Proceedings, 1995, 8 pages. Hyperbolic Space'. Proceedings of the ACM Symposium on User Kaeppner et al., “Architecture of HeiPhone: A Testbed for Audio/ Interface Software and Technology, Nov. 1994, pp. 13-14. Video Teleconferencing”, IBM European Networking Center, 1993. Lamping et al., “Visualizing Large Trees Using the Hyperbolic Kamba et al., “Using Small Screen Space More Efficiently”. CHI '96 Browser'. Apple Inc., Video Clip, MIT Media Library, on a CD, Proceedings of the SIGCHI Conference on Human Factors in Com 1995. puting Systems, Apr. 13-18, 1996, pp. 383-390. Lantz et al., “Towards a Universal Directory Service'. Departments Kang et al., “Quality Improvement of LPC-Processed Noisy Speech of Computer Science and Electrical Engineering, Stanford Univer by Using Spectral Subtraction', IEEE Transactions on Acoustics, sity, 1985, pp. 250-260. Speech and Signal Processing, vol.37, No. 6, Jun. 1989, pp.939-942. Lantz, Keith, “An Experiment in Integrated Multimedia Conferenc Keahey et al., “Non-Linear Image Magnification”, Apr. 24, 1996, 11 ing”, 1986, pp. 267-275. pageS. Lauwers et al., “Collaboration Awareness in Support of Collabora Keahey et al., “Nonlinear Magnification Fields'. Proceedings of the tion Transparency: Requirements for the Next Generation of Shared 1997 IEEE Symposium on Information Visualization, 1997. 12 Window Systems”, CHI’90 Proceedings, 1990, pp. 303-311. pageS. Lauwers et al., “Replicated Architectures for Shared Window Sys Keahey et al., “Techniques for Non-Linear Magnification Transfor tems: A Critique”, COCS 90 Proceedings of the ACM SIGOIS and mations”, IEEE Proceedings of Symposium on Information Visual IEEE CSTC-OA conference on Office information systems, ACM ization, Oct. 1996, pp. 38-45. SIGOIS Bulletin, 1990, pp. 249-260. Keahey et al., “Viewing Text With Non-Linear Magnification: An Lazzaro, Joseph J., “Adapting Desktop Computers to Meet the Needs Experimental Study”. Department of Computer Science, Indiana of Disabled Workers is Easier ThanYou Might Think”. Computers for University, Apr. 24, 1996, pp. 1-9. the Disabled, BYTE Magazine, Jun. 1993, 4 pages. Kennedy, P.J., “Digital Data Storage Using Video Disc'. IBM Tech Leahy et al., “Effect of Touch Screen Target Location on User Accu nical Disclosure Bulletin, vol. 24, No. 2, Jul. 1981, p. 1171. racy”. Proceedings of the Human Factors Society 34th Annual Meet Kerr, "An Incremental String Search in C: This DataMatching Algo ing, 1990, 5 pages. rithm Narrows the Search Space with each Keystroke’. Computer Lee, Kai-Fu, "Automatic Speech Recognition', 1989, 14 pages Language, vol. 6, No. 12, Dec. 1989, pp. 35-39. (Table of Contents). Abut et al., “Vector Quantization of Speech and Speech-Like Leung et al., “A Review and Taxonomy of Distortion-Oriented Pre Waveforms'. (IEEE Transactions on Acoustics, Speech, and Signal sentation Techniques'. ACM Transactions on Computer-Human Processing, Jun. 1982), as reprinted in Vector Quantization (IEEE Interaction (TOCHI), vol. 1, No. 2, Jun. 1994, pp. 126-160. Press, 1990), 1990, pp. 258-270. Levinson et al., “Speech synthesis in telecommunications”, IEEE Kim, E.A. S., “The Structure and Processing of Fundamental Fre Communications Magazine, vol. 31, No. 11, Nov. 1993, pp. 46-53. quency Contours'. University of Cambridge, Doctoral Thesis, Apr. Lewis, “Speech synthesis in a computer aided learning environ 1987, 378 pages. ment”, UK IT. Mar. 19-22, 1990, pp. 294-298. US 9,318.108 B2 Page 45

(56) References Cited Microsoft Corporation, Microsoft Office Word 2003 (SP2), Microsoft Corporation, SP3 as of 2005, pp. MSWord 2003 Figures OTHER PUBLICATIONS 1-5, 1983-2003. Microsoft Corporation, “Microsoft MS-DOS Operating System Lewis, Peter, “Two New Ways to Buy Your Bits', CNN Money, User's Guide”. Microsoft Corporation, 1982, pp. 4-1 to 4-16, 5-1 to available at , Dec. 31, 2003, 4 pages. Microsoft Press, "Microsoft Windows User's Guide for the Windows Lieberman, Henry, "A Multi-Scale, Multi-Layer, Translucent Virtual Graphical Environment', version3.0, 1985-1990, pp. 33-41 & 70-74. Space'. Proceedings of IEEE Conference on Information Visualiza Microsoft Windows XP. “Magnifier Utility”, Oct. 25, 2001, 2 pages. tion, Aug. 1997, pp. 124-131. Microsoft Word 2000 Microsoft Corporation, pp. MSWord Figures Lieberman, Henry, “Powers of Ten Thousand: Navigating in Large 1-5, 1999. Information Spaces'. Proceedings of the ACM Symposium on User Microsoft/Ford, “Basic Sync Commands', www.SyncMyRide.com, Interface Software and Technology, Nov. 1994, pp. 1-2. Sep. 14, 2007, 1 page. Lyon, R., “A Computational Model of Binaural Localization and Milner, N. P., “A Review of Human Performance and Preferences Separation”. Proceedings of IEEE International Conference on with Different Input Devices to Computer Systems'. Proceedings of Acoustics, Speech and Signal Processing, Apr. 1983, pp. 1148-1151. the Fourth Conference of the British Computer Society on People and Lyons et al., “Augmenting Conversations Using Dual-Purpose Computers, Sep. 5-9, 1988, pp. 341-352. Speech'. Proceedings of the 17th Annual ACM Symposium on User Miniman, Jared, "Applian Software's Replay Radio and Player interface Software and Technology, 2004, 10 pages. v1.02', pocketnow.com—Review, available at , Jul. 31, 2001, 16 pages. ings of the Twenty-Fifth Asilomar Conference on Signals, Systems Moberg et al., “Cross-Lingual Phoneme Mapping for Multilingual and Computers, Nov. 4-6, 1991, pp. 785-789. Synthesis Systems'. Proceedings of the 8th International Conference Mackenzie et al., “Alphanumeric Entry on Pen-Based Computers'. on Spoken Language Processing, Jeju Island, Korea, International Journal of Human-Computer Studies, vol. 41, 1994, pp. INTERSPEECH 2004, Oct. 4-8, 2004, 4 pages. 775-792. Moberg, M.. “Contributions to Multilingual Low-Footprint TTS Sys MacKinlay et al., “The Perspective Wall: Detail and Context tem for Hand-Held Devices'. Doctoral Thesis, Tampere University Smoothly Integrated”, ACM, 1991, pp. 173-179. of Technology, Aug. 17, 2007, 82 pages. Macsimum News, "Apple Files Patent for an Audio Interface for the Mobile Tech News, "T9 Text Input Software Updated', available at iPod', available at , archive? apple files patent for an audio interface for the Nov. 23, 2004, 4 pages. ipod, retrieved on Jul. 13, 2006, 8 pages. Mok et al., “Media Searching on Mobile Devices”, IEEE EIT 2007 Mactech, “KeyStrokes 3.5 for Mac OS X Boosts Word Prediction”, Proceedings, 2007, pp. 126-129. available at . Morland, D. V., “Human Factors Guidelines for Terminal Interface Design”. Communications ofthe ACM vol. 26, No. 7, Jul. 1983, pp. retrieved on Jan. 7, 2008, 3 pages. 484-494. Mahedero et al., “Natural Language Processing of Lyrics”. In Pro Morris et al., “Andrew: A Distributed Personal Computing Environ ceedings of the 13th Annual ACM International Conference on Mul ment”. Communications of the ACM, (Mar. 1986); vol. 29 No. 3 Mar. timedia, ACM, Nov. 6-11, 2005, 4 pages. 1986, pp. 184-201. Marcus et al., “Building a Large Annotated Corpus of English: The Muller et al., “CSCW'92 Demonstrations”, 1992, pp. 11-14. Penn Treebank”. Computational Linguistics, vol. 19, No. 2, 1993, pp. Musicmatch, “Musicmatch and Xing Technology Introduce 313-330. Musicmatch Jukebox”, Press Releases, available at . May 18, 1998, 2 pages. Masui, Toshiyuki, “POBox: An Efficient Text Input Method for Muthesamy et al., “Speaker-Independent Vowel Recognition: Handheld and Ubiquitous Computers'. Proceedings of the 1st Inter Spectograms versus Cochleagrams”, IEEE, Apr. 1990. national Symposium on Handheld and Ubiquitous Computing, 1999, My Cool Aids, “What's New', available at , 2012, 1 page. Matsuiet al., “Speaker Adaptation of Tied-Mixture-Based Phoneme Myers, Brad A., “Shortcutter for Palm', available at , retrieved on national Conference on Acoustics, Speech and Signal Processing, Jun. 18, 2014, 10 pages. Apr. 19-22, 1994, 1-125-1-128. Nadoliet al., “Intelligent Agents in the Simulation of Manufacturing Matsuzawa, A., “Low-Voltage and Low-Power Circuit Design for Systems'. Proceedings of the SCS Multiconference on AI and Simu Mixed Analog/Digital Systems in Portable Equipment', IEEE Jour lation, 1989, 1 page. nal of Solid-State Circuits, vol. 29, No. 4, 1994, pp. 470-480. Nakagawa et al., “Unknown Word Guessing and Part-of-Speech Tag Mellinger, David K., “Feature-Map Methods for Extracting Sound ging Using Support Vector Machines”. Proceedings of the 6th Frequency Modulation', IEEE Computer Society Press, 1991, pp. NLPRS, 2001, pp. 325-331. 795-799. Adium, "About Adium—Adium X Trac', available at , retrieved on Nov. 25, 2011, 2 pages. Menta, Richard, “1200 Song MP3 Portable is a Milestone Player', NCIP, "NCIP Library: Word Prediction Collection', available at available at , 1998, 4 pages. html>, Jan. 11, 2000, 4 pages. NCIP, “What is Word Prediction?”, available at , 1998, 2 pages. of California, Aug. 1987, pp. 13-19. NCIP Staff, “Magnification Technology', available at , 1994, 6 pages. dow Manager/Virtual Terminal Systems'. Department of Computer Newton, Harry, “Newton's Telecom Dictionary, Mar. 1998, pp. 62, Science, Brown University, 1981, pp. 180-189. 155, 610-61 1,771. Miastkowski, Stan, "paperWorks Makes Paper Intelligent”. Byte Nguyen et al., “Generic Manager for Spoken Dialogue Systems”. In Magazine, Jun. 1992. DiaBruck: 7th Workshop on the Semantics and Pragmatics of Dia Microsoft, “Turn on and Use Magnifier', available at , retrieved on Jun. 6, 2009. Beginners”. Computer Shopper, Feb. 1, 1992, 2 pages. US 9,318.108 B2 Page 46

(56) References Cited International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2013/047668, mailed on Jan. 8, OTHER PUBLICATIONS 2015, 13 pages. International Search Report and Written Opinion received for PCT Noik, Emanuel G., “Layout-Independent Fisheye Views of Nested Patent Application No. PCT/US2013/047668, mailed on Feb. 13, Graphs. IEEE Proceedings of Symposium on Visual Languages, 2014, 17 pages. 1993, 6 pages. International Preliminary Report on Patentability received for PCT Nonhoff-Arps et al., “StraBenmusik. Portable MP3-Spieler mit USB Patent Application No. PCT/US2013/052558, mailed on Feb. 12, Anschluss', CT Magazin Fuer Computer Technik, Verlag Heinz 2015, 12 pages. Heise GMBH, Hannover DE, No. 25, 2000, pp. 166-175. International Preliminary Report and Written Opinion received for Northern Telecom, “Meridian Mail PC User Guide', 1988, 17 pages. PCT Patent Application No. PCT/US2013/052558, mailed on Jan. Notenboom, Leo A., “Can I Retrieve Old MSN Messenger Conver 30, 2014, 15 pages. sations?', available at (http://ask-leo.com/can i retrieve old International Preliminary Report on Patentability received for PCT msn messenger conversations.html>, Mar. 11, 2004, 23 pages. Patent Application No. PCT/US2013/058916, mailed on Mar. 19, O'Connor, Rory J., “Apple Banking on Newton's Brain', San Jose 2015, 8 pages. Mercury News, Apr. 22, 1991. International Preliminary Report on Patentability received for PCT Ohsawa et al., “A computational Model of an Intelligent Agent Who Patent Application No. PCT/US2013/060121, mailed on Apr. 2, Talks with a Person'. Research Reports on Information Sciences, 2015, 6 pages. Series C. No. 92, Apr. 1989, pp. 1-18. International Search Report and Written Opinion received for PCT Ohtomo et al., “Two-Stage Recognition Method of Hand-Written Patent Application No. PCT/US2013/060121, mailed on Dec. 6, Chinese Characters. Using an Integrated Neural Network Model”, 2013, 8 pages. Denshi Joohoo Tsuushin Gakkai Ronbunshi, D-II, vol. J74, Feb. International Search Report and Written Opinion received for PCT 1991, pp. 158-165. Patent Application No. PCT/US2014/040961, mailed on Mar. 10, Okazaki et al., “Multi-Fisheye Transformation Method for Large 2015, 5 pages. Scale NetworkMaps', IEEE Japan, vol. 44, No. 6, 1995, pp. 495-500. Invitation to Pay Additional Fees received for PCT Application No. Adobe.com, “Reading PDF Documents with Adobe Reader 6.0 A PCT/US2014/040961, mailed on Jan. 14, 2015, 3 pages. Guide for People with Disabilities”. Available online at "https:// AppleEvent Manager, which is described in the publication Inside www.adobe.com/enterprise? accessibility/pdfs/acro6 cg ue.pdf. Macintosh vol. VI, available from Addison-Wesley Publishing Com Jan. 2004, 76 pages. pany, 1985. Bertolucci, Jeff, “Google Adds Voice Search to Chrome Browser'. Dual Rate Speech Coder for Multimedia Communications Transmit PC World, Jun. 14, 2011, 5 pages. ting at 5.3 and 6.3 kbit/s, International Telecommunication Union Dobrisek et al., “Evolution of the Information-Retrieval System for Recommendation G.723, 7 pages. Blind and Visually-Impaired People'. International Journal of N200 Hands-Free Bluetooth Car Kit, available at

(56) References Cited Mangu et al., “Finding Consensus in Speech Recognition: Word Error Minimization and Other Applications of Confusion Networks', OTHER PUBLICATIONS Computer Speech and Language, vol. 14, No. 4, 2000, pp. 291-294. Manning et al., “Foundations of Statistical Natural Language Pro Fiscus, J. G., “A Post-Processing System to Yield Reduced Word cessing”. The MIT Press, Cambridge Massachusetts, 1999, pp. Error Rates: Recognizer Output Voting Error Reduction (ROVER)'. 10-11. IEEE Proceedings, Automatic Speech Recognition and Understand Meng et al., “Generating Phonetic Cognates to Handle Named Enti ing, Dec. 14-17, 1997, pp. 347-354. ties in English-Chinese Cross-Language Spoken Document Gonnet et al., “Handbook of Algorithms and Data Structures: in Retrieval”. Automatic Speech Recognition and Understanding, 2001, Pascal and C. (2nd ed.)”. Addison-Wesley Longman Publishing Co., pp. 311-314. 1991, 17 pages. Miller, Chance, “Google Keyboard Updated with New Personalized Gruber et al., U.S. Appl. No. 61/186,414, filed Jun. 12, 2009 titled Suggestions Feature', available at , Mar. 19, 2014, 4 pages. "Generating and Processing Data Items That Represent Tasks to International Preliminary Report on Patentability received for PCT Perform'. 68 pages. Patent Application No. PCT/GB2009/051684, mailed on Jun. 23, Gruber et al., U.S. Appl. No. 61/657,744, filed Jun. 9, 2012 titled 2011, 10 pages. "Automatically Adapting User Interfaces for Hands-Free Interac International Search Report received for PCT Patent Application No. tion' 40 pages. PCT/GB2009/051684, mailed on Mar. 12, 2010, 4 pages. Gruber al., U.S. Appl. No. 07/976,970, filed Nov. 16, 1992 titled International Preliminary Examination Report on received for PCT "Status Bar for Application Windows'. Patent Application No. PCT/US 1993/12637, mailed on Apr. 10, Guay, Matthew, “Location-Driven Productivity with Task Ave'. 1995, 7 pages. available at shttp://iphone.appstorm.net/reviews/productivity/loca International Preliminary Report on Patentability received for PCT tion-driven-productivity-with-task-avef>, Feb. 19, 2011, 7 pages. Patent Application No. PCT/US2009/051954, issued on Mar. 24, Guim, Mark, “How to Seta Person-Based Reminder with Cortana', 2011, 8 pages. available at (http://www.wpcentral.com/how-to-person-based-re International Search Report and Written Opinion received for PCT minder-cortana, Apr. 26, 2014, 15 pages. Haitsma et al., “A Highly Robust Audio Fingerprinting System'. In Patent Application No. PCT/US2009/051954, mailed on Oct. 30, Proceedings of the International Symposium on Music Information 2009, 10 pages. Retrieval (ISMIR), 2002, 9 pages. International Preliminary Report on Patentability received for PCT Hendrickson, Bruce, “Latent Semantic Analysis and Fiedler Patent Application No. PCT/US2012/040571, mailed on Dec. 19, Retrieval”. Discrete Algorithms and Mathematics Department, 2013, 10 pages. Sandia National Labs, Albuquerque, NM, Sep. 21, 2006, 12 pages. International Preliminary Report on Patentability received for PCT Henrich et al., “Language Identification for the Automatic Graph Patent Application No. PCT/US2005/030234, issued on Mar. 20, eme-To-Phoneme Conversion of Foreign Words in A German Text 2007.9 pages. To-Speech System”. Proceedings of the European Conference on International Search Report and Written Opinion received for PCT Speech Communication and Technology, vol. 2, Sep. 1989, pp. 220 Patent Application No. PCT/US2005/030234, mailed on Mar. 17, 223. 2006, 11 pages. ID3.org, “id3v2.4.0—Frames', available at , retrieved on Jan. 22, 2015, 41 pages. Patent Application No. PCT/US2012/040801, mailed on Dec. 19, Jawaid et al., “Machine Translation with Significant Word Reorder 2013, 16 pages. ing and Rich Target-Side Morphology”. WDS 11 Proceedings of International Search Report and Written Opinion received for PCT Contributed Papers, Part I, 2011, pp. 161-166. Patent Application No. PCT/US2012/040801, mailed on Oct. 22, Jiang et al., “A Syllable-based Name Transliteration System”. Pro 2012, 20 pages. ceedings of the 2009 Named Entities Workshop, Aug. 7, 2009, pp. Amano, Junko, "A User-Friendly Authoring System for Digital Talk 96-99. ing Books', IEICE Technical Report, The Institute of Electronics, Kane et al., “Slide Rule: Making Mobile Touch Screens Accessible to Information and Communication Engineers, vol. 103 No. 418, Nov. Blind People Using Multi-Touch Interaction Techniques'. ASSETS, 6, 2003, pp. 33-40. Oct. 13-15, 2008, pp. 73-80. International Search Report and Written Opinion received for PCT Kazemzadeh et al., "Acoustic Correlates of User Response to Error in Patent Application No. PCT/US2012/043100, mailed on Nov. 15, Human-Computer Dialogues'. Automatic Speech Recognition and 2012, 8 pages. Understanding, 2003, pp. 215-220. International Preliminary Report on Patentability received for PCT Kikui, Gen-Itiro, “Identifying the Coding System and Language of Patent Application No. PCT/US2012/056382, mailed on Apr. 10, On-Line Documents on the Internet'. International Conference on 2014, 9 pages. Computational, Aug. 1996, pp. 652-657. International Preliminary Report on Patentability received for PCT Kohler, Joachim, “Multilingual Phone Models for Vocabulary-Inde Patent Application No. PCT/US2013/0284.12, mailed on Sep. 12, pendent Speech Recognition Tasks'. Speech Communication, vol. 2014, 12 pages. 35, No. 1-2, Aug. 2001, pp. 21-30. International Search Report and Written Opinion received for PCT Kroon et al., “Pitch Predictors with High Temporal Resolution'. Patent Application No. PCT/US2013/0284.12, mailed on Sep. 26, IEEE, vol. 2, 1990, pp. 661-664. 2013, 17 pages. Ladefoged, Peter, "A Course in Phonetics', New York, Harcourt, International Preliminary Report on Patentability received for PCT Brace, Jovanovich, Second Edition, 1982. Patent Application No. PCT/US2013/028920, mailed on Sep. 18, Lau et al., “Trigger-Based Language Models: A Maximum Entropy 2014, 11 pages. Approach”, ICASSP’93 Proceedings of the 1993 IEEE international International Search Report and Written Opinion received for PCT conference on Acoustics, speech, and signal processing: speech pro Patent Application No. PCT/US2013/028920, mailed on Jun. 27. cessing vol. II, 1993, pp. 45-48. 2013, 14 pages. Lee et al., "On URL Normalization'. Proceedings of the International International Preliminary Report on Patentability received for PCT Conference on Computational Science and its Applications, ICCSA Patent Application No. PCT/US2013/029156, mailed on Sep. 18, 2005, pp. 1076-1085. 2014, 7 pages. Leveseque et al., “A Fundamental Tradeoff in Knowledge Represen International Search Report and Written Opinion received for PCT tation and Reasoning. Readings in Knowledge Representation Patent Application No. PCT/US2013/029156, mailed on Jul. 15, 1985, 30 pages. 2013, 9 pages. US 9,318.108 B2 Page 48

(56) References Cited Stuker et al., “Cross-System Adaptation and Combination for Con tinuous Speech Recognition: The Influence of Phoneme Set and OTHER PUBLICATIONS Acoustic Front-End'. Influence of Phoneme Set and Acoustic Front End, Interspeech, Sep. 17-21, 2006, pp. 521-524. International Preliminary Report on Patentability received for PCT Patent Application No. PCT/US2013/041233, issued on Nov. 18, Sundaramet al., “Latent Perceptual Mapping with Data-Driven Vari 2014, 8 pages. able-Length Acoustic Units for Template-Based Speech Recogni International Search Report received for PCT Patent Application No. tion”, ICASSP 2012, Mar. 2012, pp. 4125-4128. PCT/US2013/041233, mailed on Nov. 22, 2013, 3 pages. Viegas et al., “Chat Circles'. SIGCHI Conference on Human Factors International Search Report and Written Opinion received for PCT in Computing Systems, May 15-20, 1999, pp. 9-16. Patent Application No. PCT/US2013/058916, mailed on Sep. 8, Waibel, Alex, “Interactive Translation of Conversational Speech'. 2014, 10 pages. Computer, vol. 29, No. 7, Jul. 1996, pp. 41-48. International Search Report and Written Opinion received for PCT Wang et al., “An Industrial-Strength Audio Search Algorithm”. In Patent Application No. PCT/US2014/015418, mailed on Aug. 26. Proceedings of the International Conference on Music Information 2014, 17 pages. Retrieval (ISMIR), 2003, 7 pages. International Search Report and Written Opinion received for PCT Amano et al., “A User-friendly Multimedia Book Authoring Sys Patent Application No. PCT/US2014/028785, mailed on Oct. 17, 2014, 23 pages. tem'. The Institute of Electronics, Information and Communication International Search Report and Written Opinion received for PCT Engineers Technical Report, vol. 103, No. 416, Nov. 2003, pp. 33-40. Patent Application No. PCT/US2014/029050, mailed on Jul. 31, Glass, J., et al., “Multilingual Spoken-Language Understanding in 2014.9 pages. the MIT Voyager System.” Aug. 1995, http://groups.csail.mit.edu/ International Search Report and Written Opinion received for PCT sls/publications/1995/speechcomm95-voyager.pdf, 29 pages. Patent Application No. PCT/US2014/029562, mailed on Sep. 18, Goddeau, D., et al., “A Form-Based Dialogue Manager for Spoken 2014, 21 pages. Language Applications.” Oct. 1996, http://phasedance.com/pdf International Search Report and Written Opinion received for PCT ics.lp96.pdf. 4 pages. Patent Application No. PCT/US2014/040401, mailed on Sep. 4, Goddeau, D., et al., “Galaxy: A Human-Language Interface to On 2014, 10 pages. Line Travel Information.” 1994 International Conference on Spoken International Search Report and Written Opinion received for PCT Language Processing, Sep. 18-22, 1994, Pacific Convention Plaza Patent Application No. PCT/US2014/040403, mailed on Sep. 23, 2014.9 pages. Yokohama, Japan, 6 pages. International Search Report and Written Opinion received for PCT Meng, H., et al., “Wheels: A Conversational System in the Automo Patent Application No. PCT/US2014/041159, mailed on Sep. 26, bile Classified Domain.” Oct. 1996, httphttp://citeseerx.ist.psu.edu/ 2014, 10 pages. viewdoc/summary?doi=10.1.1.16.3022, 4 pages. International Search Report and Written Opinion received for PCT Phoenix Solutions, Inc. v. West Interactive Corp., Document 40, Patent Application No. PCT/US2014/041173, mailed on Sep. 10, Declaration of Christopher Schmandt Regarding the MIT Galaxy 2014, 11 pages. System dated Jul. 2, 2010, 162 pages. International Search Report and Written Opinion received for PCT Seneff, S., et al., “A New Restaurant Guide Conversational System: Patent Application No. PCT/US2014/049568, mailed on Nov. 14, Issues in Rapid Prototyping for Specialized Domains.” Oct. 1996, 2014, 12 pages. citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.16... rep.... 4 International Search Report and Written Opinion received for PCT pageS. Patent Application No. PCT/US2014/23822 mailed on Sep. 25, 2014, 14 pages. Vlingo InCar, “Distracted Driving Solution with Vlingo InCar” 2:38 Roddy et al., “Interface Issues in Text Based Chat Rooms'. SIGCHI minute video uploaded to YouTube by Vlingo Voice on Oct. 6, 2010, Bulletin, vol. 30, No. 2, Apr. 1998, pp. 119-123. http://www.youtube.com/watch?v=Vcs8XfXxg74, 2 pages. Rose et al., “Inside Macintosh'', vols. I, II, and III. Addison-Wesley Zue, V., "Conversational Interfaces: Advances and Challenges.” Sep. Publishing Company, Inc., Jul. 1988, 1284 pages. 1997, http://www.cs.cmu.edu/~dod/papers/Zue97.pdf, 10 pages. Russo et al., “Urgency is a Non-Monotonic Function of Pulse Rate'. Zue, V. W. “Toward Systems that Understand Spoken Language.” Journal of the Acoustical Society of America, vol. 122, No. 5, 2007. Feb. 1994, ARPA Strategic Computing Institute, (C) 1994 IEEE, 9 6 pages. pageS. Sankar, Ananth, “Bayesian Model Combination (BAYCOM) for Final office action in U.S. Appl. No. 1 1/518,292, mailed on May 10, Improved Recognition', IEEE International Conference on Acous 2011. tics, Speech, and Signal Processing (ICASSP), Mar. 18-23, 2005, pp. Applicant's office action response in U.S. Appl. No. 1 1/518,292, 845-848. Schone et al., “Knowledge-Free Induction of Morphology Using mailed on Sep. 6, 2011. Latent Semantic Analysis”. Proceedings of the 2nd Workshop on David Roddy, Leo Orbst, Adam Cheyer; "Communication and Col Learning Language in Logic and the 4th Conference on Computa laboration in a Landscape of B2B eMARKETPLACE, Jun. 2001. tional Natural Language Learning, vol. 7, 2000, pp. 67-72. Adam Cheyer; “A Perspective on AI & Agent Technologies for Sethy et al., “A Syllable Based Approach for Improved Recognition SCM, 2001. of Spoken Names', ITRW on Pronunciation Modeling and Lexicon Adam Cheyer; “Active: A unified platform for building intelligent Adaptation for Spoken language Technology (PMLA2002), Sep. web interaction assistants'; Dec. 18, 2006. 14-15, 2002, pp. 30-35. WSMO Working Group: John Domingue, Dumitru Roman, Michael Stifleman, L. “Not Just Another Voice Mail System”. Proceedings of Stollberg: “Web Service Modeling Ontology (WSMO)”; Jun. 2005. 1991 Conference, American Voice, Sep. 24-26, 1991, Atlanta GA., Bussler et al: “Web Service Execution Environment (WSMX)”; Jun. Sep. 1991, pp. 21-26. 3, 2005. Strom et al., “Intelligent Barge-In in Conversational Systems', MIT laboratory for Computer Science, 2000, 4 pages. * cited by examiner U.S. Patent Apr. 19, 2016 Sheet 1 of 47 US 9,318,108 B2

Z-1002 Intelligent Automated Assistant

Active Input Short Term Long Term Elicitation Personal Personal O94. Memory 1052 Memory 1054

Task FOW Models O86 Output to User input User OO4. Vocabulary 1008 (58 Ontology Dialog Flow Models

Language 1087

Pattern Recognizers

OSO Other Other Events/FactS Actions

Dialog Flow OO6 PrOCeSSOr OC Domain O8O Entity Databases Output O72 Processor

Services O90 Orchestration Language O82 Interpreter

Services 1084

FIG. 1 U.S. Patent Apr. 19, 2016 Sheet 2 of 47 US 9,318,108 B2

::::ik& 8:::::::::::::::388 fixir iaiiar fox 8ear my office :::::: Pastaia Ristorate & Engieces ; :::::::::8:38: ::::::::::888. isis St. 3:38. A SSIS

Spiecia Ristorate

FIG. 2 U.S. Patent Apr. 19, 2016 Sheet 3 of 47 US 9,318,108 B2

6O \ 68Y

INTERFACE(S)

61 MEMORYa 62 MEMORY -65 -59 63 PROCESSOR(S) u? - 57 -55 67

FIG. 3 U.S. Patent Apr. 19, 2016 Sheet 4 of 47 US 9,318,108 B2

Intelligent Automated Input device ASSistant 12O6 OO2 Output device 12O7

Storage device 1208

Memory Processor(s) 63 21 O

FIG. 4 U.S. Patent Apr. 19, 2016 Sheet 5 of 47 US 9,318,108 B2

ClientS 1304 Servers 1340

( Network s 1361

External Services 1360

FIG. 5 U.S. Patent Apr. 19, 2016 Sheet 6 of 47 US 9,318,108 B2

Computer I/O

devices and Client on Web BrOWSer SeSOS

Mobile Device Client on Mobil ServerS 340

I/O and lent on Mobile = TIT U

SeSOS Device AO6 3O4. E.

Consumer Client on Consumer Y Appliance I/O Appliance and SensOrS 3O4C AO 1 NetWork s 1361 s

Automobile Client in embedded

Dashboard system and Sensors (e.g. automobile) A 4. 3O4.D

External Services Networked 36O Computing I/O Client on any other devices and networked Computing device SeSOS 48 3O4.

Email Client Email Modality Serve

Messaging Modality Instant Messaging Server Client 428 FIG. 6

Voice telephone U.S. Patent Apr. 19, 2016 Sheet 7 of 47 US 9,318,108 B2

Subset of Cache of Cache of Language Short Term Long Term Subset of Patter Personal Personal Vocabulary Recognizers Memory Memory 333 OSO a 52a. 5-3

Client part of Input Elicitation 1094a Client part of Output Processing 1092a

340

Server part of Input Elicitation 1094b Server part of Output Processing 1092b

Full Library of Master of Master of Language Short Tern Long Term AI Patter Personal Personal Vocabulary Recognizers Memory Memory O58 O6C 52 O54

Language Dialog Flow Output Interpreter Processor Processor 1 O70 3.

Domain Task Flow Services Entity Models Orchestration Databases

External Services 1360

FIG. 7 U.S. Patent Apr. 19, 2016 Sheet 8 of 47 US 9,318,108 B2

O72 Active Ontology / Domain Entity Databases Domain Models

| Restaurant 1610 ... Restaurants Dining Out DB 552 Domain Model 622 Business LOCation 63 Name 62

Cuisines Served Task FOW Models Vocabulary Databases 65 Event Planning Task Model 630 Meal Event 66

Party 1617

Dialog Flow Models Service Models

get Constraints required for restaurant transaction reservation Service 672

FIG. 8 U.S. Patent Apr. 19, 2016 Sheet 9 of 47 US 9,318,108 B2

A OO2 Intelligent Automated Assistant

Active input Elicitation 1094 User input

OO4. Active Ontology 1050

Domain Models Short Term PerSona

O56 Memory 1052

Other Vocabulary 1058 Long Term Personal EvertSFactS Memory 1054 OO6 Language Pattern Recognizers 1060

Language Dialog Flow Output Processor Interpreter 1070 ProCeSSOr 108O O90 Output to User O08

Domain Entity Dialog Flow Task FOW Services Databases 1072 Models 1087 Models 1086 Orchestration

Service Models

Services 1084

FIG. 9 U.S. Patent Apr. 19, 2016 Sheet 10 of 47 US 9,318,108 B2

Active input Elicitation Procedure Start 20

System offers input Channels 2

User selects input Channel 22

System Suggests options that are relevant prior to input 23

System Suggests options that are options that are relevant for Current offered 24 input 29

User selects an option that is offered? 25

Preference: autoSubmit option? 30 User enters partial Add latest input to input 26 Cumulative input 28

User indicates input is complete? 27 yes return Current input

FIG 10 U.S. Patent Apr. 19, 2016 Sheet 11 of 47 US 9,318,108 B2

Active Typed-Input Elicitation Procedure Start 110

Receive Partial Text input from device 111

User indicates text is complete? 112

O

Suggest Possible Options 114 Annotated Text input 119

Candidate Suggestions 116 return

User picks a suggestion? 118

Transform Current input to include selected suggestion 117

FIG. 11 U.S. Patent Apr. 19, 2016 Sheet 12 of 47 US 9,318,108 B2

120

&

1202 U.S. Patent Apr. 19, 2016 Sheet 13 of 47 US 9,318,108 B2

301

Cent e

FIG. 13 1306 U.S. Patent Apr. 19, 2016 Sheet 14 of 47 US 9,318,108 B2

40

1305

303

at SS g att

1304

FIG. 14 1306 U.S. Patent Apr. 19, 2016 Sheet 15 Of 47 US 9,318,108 B2

501

35

, , , , , , ,

, , , , , , , '.

, , , , , , , ,

, ,

& 8 , , ,

, ,

8, , , ,

, , , , , , , , ,

F.G. 15 1306 U.S. Patent Apr. 19, 2016 Sheet 16 of 47 US 9,318,108 B2

1305

203 602 if he Fire

F.G. 16 1306 U.S. Patent Apr. 19, 2016 Sheet 17 of 47 US 9,318,108 B2

1701

1305 1602 Nhat is showing at the The Fillmore ... by Category concerts, sports, festivals X

... in City or zipcode x

... named even name X ... featuring performer name

:

FIG. 17 1306 U.S. Patent Apr. 19, 2016 Sheet 18 of 47 US 9,318,108 B2

801

305

602

f t

...

SS S S 1304

FIG. 18 306 U.S. Patent Apr. 19, 2016 Sheet 19 Of 47 US 9,318,108 B2

1305

1602

FIG. 19 1306 U.S. Patent Apr. 19, 2016 Sheet 20 of 47 US 9,318,108 B2

what is showing at the The Fiore r is weeker

2003 'm looking for events at for 8 exei U.S. Patent Apr. 19, 2016 Sheet 21 of 47 US 9,318,108 B2

2102

2103

FIG 21 U.S. Patent Apr. 19, 2016 Sheet 22 of 47 US 9,318,108 B2

Active Speech Input Elicitation Procedure Start 221

Receive Speech input 121

Speech-to-text Service 122

Candidate interpretations 24

Rank interpretations by Semantic relevance 126

Present Top interpretation

POSsible above threshold? Interpretations 128 132

Automatically Candidate select input interpretations 13O 134

User picks an interpretation

Annotated Text input

FIG. 22 U.S. Patent Apr. 19, 2016 Sheet 23 of 47 US 9,318,108 B2

Active GUI-based Input Elicitation Start 140

Present GU with links and buttons 14.

User interacts With GU element 142

data from link or button 144

COnvert to uniform format 146

return

FIG. 23 U.S. Patent Apr. 19, 2016 Sheet 24 of 47 US 9,318,108 B2

Active Dialog Suggestion input Elicitation Start 150

Suggest POSSible Responses 151

Suggested Responses 152

User picks a Suggested response 154

COnvert to uniform format 154.

return

FIG. 24 U.S. Patent Apr. 19, 2016 Sheet 25 Of 47 US 9,318,108 B2

Active Monitoring for Relevant Events Procedure Start 160

Monitor for Relevant Events 16

POSSible Event Triggers 162

Event SOrted and filtered by relevance and match to models 164

COnvert event data to uniform input format 166

return

FIG. 25 U.S. Patent Apr. 19, 2016 Sheet 26 of 47 US 9,318,108 B2

Active Multimodal input Elicitation Procedure Start 100

Actively Elicit Typed Actively Elicit Actively Present Actively offer Actively Monitor Input 260 Speech input 2620 GUI for input Suggested for Relevant 264. Possible Asynchronous Responses in Events 268) Dialogs 2650

Uniform Annotated input Format

FIG. 26 U.S. Patent Apr. 19, 2016 Sheet 27 Of 47 US 9,318,108 B2

liardarin Gourmet Cupertino

S8 kg 3x3 83rd Ciper;inx, CA 938.4 :3:33 R

FIG. 27 U.S. Patent Apr. 19, 2016 Sheet 28 of 47 US 9,318,108 B2

Natural Language Processing Procedure Start 200

Language Input 202 language pattern recognizers | Word/phrase

1060 w Matching 210 (vs.

Vocabulary y M personal database 1058 A. w memory candidate syntactic parSeS 212 perSonal memory O54. active Ontology Semantic K with domain and matching 220 task models 1050

Candidate Semantic parSeS 222

Disambiguation 230

Filter, Sort 232

Representation of User Intent (parses) 290 2 FIG. 28 U.S. Patent Apr. 19, 2016 Sheet 29 Of 47 US 9,318,108 B2

290 2902 who is playing this weeked at the fire 2903

wi: 8888:

FIG. 29 U.S. Patent Apr. 19, 2016 Sheet 30 of 47 US 9,318,108 B2

300 3002

?book me a table for dinner 300 e, and how many

FIG. 30 U.S. Patent Apr. 19, 2016 Sheet 31 of 47 US 9,318,108 B2

310 s 3102

Ok, here are some estaurants with open tales & 8 3103 Click a time to book a table:

Cafe Torre 304 :

Fontana's tali

FIG. 31 U.S. Patent Apr. 19, 2016 Sheet 32 of 47 US 9,318,108 B2

Dialog and Flow Analysis Procedure Start 300

Representation of User Intent (multiple NL parses) 290

ls intent too yeS ambiguous? 310

Domain

Models 1056 S preferred interpretation (Domain and Task parameters) Task FOW Models 1086

Determine next Set flow step to Dialod Flow \------w step for promprompt for more

No. 1087 task/domain/ information 322 dialog flow 320

Representation of Request: - dialog flow step - domain parameters - task parameters - natural language parse 390

FIG. 32 U.S. Patent Apr. 19, 2016 Sheet 33 of 47 US 9,318,108 B2

Automated Call and Response Procedure Start 10

Actively Elicit User input 100

Uniform Annotated input Format 2690

Process input (e.g., interpret natural language utterance) 200

Representation of User Intent (multiple) 290 2

identify Task, Task Parameters, and Dialog Flow Step 300

Execute Flow Step, Orchestrating Services 400 Services 1084

Generate Dialog Response (modality independent) 500 yeS User is done? 790 Format Response Output for Modality 600

Present Output to User 700 Gend D FIG. 33 U.S. Patent Apr. 19, 2016 Sheet 34 of 47 US 9,318,108 B2

Constrained Selection Task 351

Solicit SOme Criteria and/or Constraints 352

broWSe Refine Criteria matching & Constraints 354 itemS 353

Select item 355

Follow-on tasks 359

Book 356 Remember 357 Share 358

FIG. 34 U.S. Patent Apr. 19, 2016 Sheet 35 of 47 US 9,318,108 B2

350

Outback Steakhouse

Pick Up Stix

Gochi Japanese Fusion Tapas U.S. Patent Apr. 19, 2016 Sheet 36 of 47 US 9,318,108 B2

360

Gochi Japanese Fusion Tapas

19980 E Homestead Rd Cupertino, CA 95.014

Share a : 8883:

FIG. 36 U.S. Patent Apr. 19, 2016 Sheet 37 Of 47 US 9,318,108 B2

Flow and Service Orchestration Procedure Start 400

Representation of Request 390

yes Services Delegation Execute Standalone Required? 402 flow step 403

Determine which

Services can meet request 404 Determine which Services can annotate Current results 422 Invoke Services 450

Invoke Services

Results from 450 multiple services 41 O

Results from multiple services Validate and 424 Merge 412

Relax request parameters M 426

Sufficientrequest? results 416 for Sortsort 428 yes

ls annotation Uniform required to answer Representation request? 418 of Response 490 O U.S. Patent Apr. 19, 2016 Sheet 38 of 47 US 9,318,108 B2

Service InVOCation PrOCedure Start 450

Representation of Reduest 390 task parameters, etc

Transform relevant task parameters to meet Service AP 452

Service Call the Service 454

MOCelS 1 O88

Map output of Service to unified result representation 456

Uniform Result representation 490

FIG. 38 U.S. Patent Apr. 19, 2016 Sheet 39 Of 47 US 9,318,108 B2

Multiphase Output Procedure Start 700

702 704

Interpret speech 710

candidate speech Paraphrase Speech interpretations 712 Interpretations 730

Interpret natural language utterance 74

Representation of Paraphrase Text User Intent 716 Interpretations 732

Identify Task, Task Parameters, Paraphrase Task and and Dialog Flow Step 718 Domain Interpretations 734

Dispatch to Services, Show real time progress gather results, and of Service Coordination Combine results 720 736

Uniform Representation of Response 722

Format Response for Output Paraphrase summary or Modality 724 when answer 738

FIG. 39 U.S. Patent Apr. 19, 2016 Sheet 40 of 47 US 9,318,108 B2

4001

4002

Paraphrase Speech Interpretations 3: Francisco next Wednesday

Paraphrase Task and cs Domain Interpretatio ------: 888 m : s : arod 8: 33

FIG. 40 U.S. Patent Apr. 19, 2016 Sheet 41 of 47 US 9,318,108 B2

Paraphrase Text Interpretations 33

Show real time progress of Service coordination 3:

i388. xxxx x8 Paraphrase summary or recipitation is key for answer 38

405 FIG. 41 U.S. Patent Apr. 19, 2016 Sheet 42 of 47 US 9,318,108 B2

Multimodal Output Processing Procedure Start 600

Uniform Representation of Response 490

Device and Modality Format response Domain Data

for device and Models 610 modality 612 Models 614

Generate Text Generate Email Generate GUI Generate Speech Message Output Output Output Output 620 622 624 626

Send to Text Send as Email Send to device Or Send to speech Message Channel Message web browser for generation module 630 632 rendering 636 634

FIG. 42 U.S. Patent Apr. 19, 2016 Sheet 43 of 47 US 9,318,108 B2

4301 4302

Sis.' ' ' ' ' ' ' west' ' ' ' ' ' ' ' ' winds' ' ' ' ' ' ' ' ' ' ' ' ' ' S.' ' ' ' ' ' ' ' ' ' E.' ' ' ' ' rig......

is 88. {{i}; {x 3i: 88 cay Rpt...bg.coxing south-west 5 to 5 38; 8.3:

t doesn't look Šike rais is key for 388 day.

&: *88 38 it 8 w888 for 83&S Tuesday 67° 55° it in Kiew York, NY for 3: Fristy cicady is tie Ficting then 8 becoRirig partly cloudy, bighs is the kipper $g to frict 3s, west writis Sr. i0 rph, 8

tiesday night: 8 Rostly coucy with is test8 shoxyers : 888 sty totaly. 343 it this sigger Rorning...the Eissty starsity in the is. East is is C. afteroon, highs arouad F. east sinda, it} Eg:...bgcensing southwest 5 to 5 to Siri. Etics of Fair 33 perce:Ft. KY Eggs 8

FIG. 43A FIG. 4.3B U.S. Patent Apr. 19, 2016 Sheet 44 of 47 US 9,318,108 B2

440

4403

My Restaurants (2} My Settings (4)

FIG. 4.4B

Aexander's Steakhetise (333 N. Wolfe Rd Cupertino 408-446-222 Miyake 850 Se Anza 8x8 Cupertino FIG. 44A 408-253-2688

4404

FIG. 44C U.S. Patent Apr. 19, 2016 Sheet 45 of 47 US 9,318,108 B2

All Local SearchTypes

“N Select tes

FIG. 45 U.S. Patent Apr. 19, 2016 Sheet 46 of 47 US 9,318,108 B2

4600

\ Infer task\, and domain 4609

A " " ' " 'N'." " " " " wr - r w w we a s an a? will wh a mid- a v - M ------"...... N...... :i Constrained Selection Task 4-1 w : Solicit Criteria, constraints : 4603 -- Paraphrase Refine criteria Explain & P Sent & COnstraints - u- u 4604, 4605 4606

Select tem u- : U.S. Patent Apr. 19, 2016 Sheet 47 of 47 US 9,318,108 B2

4799 End 4701 Starta No

Yes 4717 47O2 Does user provide Receive input

4703 4705 is task known? Request clarifying input

Yes

47O6 No 4704 Proceed to is task Constrained specified task flow selection?

Yes

4707 No 4708 Can selection class be Offer choice of known determined? Selection classes

Yes

4709 No 4,710 Can required constraints be Prompt for required determined? information

Yes

No 4,712 4711 Any results? Offer ways to relax

Constraints

Yes

4713 Offer list of items

No 4716 4714 User selects item? Offer ways to select other FIG. 47 Criteria and/or Constraints Yes 4715 Perform follow-on task, if applicable US 9,318, 108 B2 1. 2 INTELLIGENT AUTOMATED ASSISTANT capabilities provided by different software components (e.g., for Supporting natural language recognition and dialog, mul CROSS-REFERENCE TO RELATED timodal input, personal information management, task flow APPLICATIONS management, orchestrating distributed services, and the like). Furthermore, to offer intelligent interfaces and useful func This application claims priority from U.S. Provisional tionality to users, the intelligent automated assistant of the Patent Application Ser. No. 61/295,774 for “Intelligent Auto present invention may, in at least some embodiments, coor mated Assistant, filed Jan. 18, 2010, which is incorporated dinate these components and services. The conversation herein by reference. interface, and the ability to obtain information and perform This application is further related to U.S. patent application 10 follow-on task, are implemented, in at least Some embodi Ser. No. 1 1/518,292 for “Method and Apparatus for Building ments, by coordinating various components such as language an Intelligent Automated Assistant, filed Sep. 8, 2006, which components, dialog components, task management compo is incorporated herein by reference. nents, information management components and/or a plural This application is further related to U.S. Provisional ity of external services. Patent Application Ser. No. 61/186,414 for "System and 15 According to various embodiments of the present inven Method for Semantic Auto-Completion', filed Jun. 12, 2009, tion, intelligent automated assistant systems may be config which is incorporated herein by reference. ured, designed, and/or operable to provide various different types of operations, functionalities, and/or features, and/or to FIELD OF THE INVENTION combine a plurality of features, operations, and applications of an electronic device on which it is installed. In some The present invention relates to intelligent systems, and embodiments, the intelligent automated assistant systems of more specifically for classes of applications for intelligent the present invention can performany or all of actively elic automated assistants. iting input from a user, interpreting user intent, disambiguat ing among competing interpretations, requesting and receiv BACKGROUND OF THE INVENTION 25 ing clarifying information as needed, and performing (or initiating) actions based on the discerned intent. Actions can Today's electronic devices are able to access a large, grow be performed, for example, by activating and/or interfacing ing, and diverse quantity of functions, services, and informa with any applications or services that may be available on an tion, both via the Internet and from other sources. Function electronic device, as well as services that are available overan ality for Such devices is increasing rapidly, as many consumer 30 electronic network such as the Internet. In various embodi devices, Smartphones, tablet computers, and the like, are able ments, such activation of external services can be performed to run software applications to perform various tasks and via APIs or by any other suitable mechanism. In this manner, provide different types of information. Often, each applica the intelligent automated assistant systems of various tion, function, website, or feature has its own user interface embodiments of the present invention can unify, simplify, and and its own operational paradigms, many of which can be 35 improve the user's experience with respect to many different burdensome to learn or overwhelming for users. In addition, applications and functions of an electronic device, and with many users may have difficulty even discovering what func respect to services that may be available over the Internet. The tionality and/or information is available on their electronic user can thereby be relieved of the burden of learning what devices or on various websites; thus, such users may become functionality may be available on the device and on web frustrated or overwhelmed, or may simply be unable to use 40 connected services, how to interface with Such services to get the resources available to them in an effective manner. what he or she wants, and how to interpret the output received In particular, novice users, or individuals who are impaired from Such services; rather, the assistant of the present inven or disabled in some manner, and/or are elderly, busy, dis tion can act as a go-between between the user and Such tracted, and/or operating a vehicle may have difficulty inter diverse services. facing with their electronic devices effectively, and/or engag 45 In addition, in various embodiments, the assistant of the ing online services effectively. Such users are particularly present invention provides a conversational interface that the likely to have difficulty with the large number of diverse and user may find more intuitive and less burdensome than con inconsistent functions, applications, and websites that may be ventional graphical user interfaces. The user can engage in a available for their use. form of conversational dialog with the assistant using any of Accordingly, existing systems are often difficult to use and 50 a number of available input and output mechanisms, such as to navigate, and often present users with inconsistent and for example speech, graphical user interfaces (buttons and overwhelming interfaces that often prevent the users from links), text entry, and the like. The system can be imple making effective use of the technology. mented using any of a number of different platforms, such as device APIs, the web, email, and the like, or any combination SUMMARY 55 thereof. Requests for additional input can be presented to the user in the context of such a conversation. Short and long term According to various embodiments of the present inven memory can be engaged so that user input can be interpreted tion, an intelligent automated assistant is implemented on an in proper context given previous events and communications electronic device, to facilitate user interaction with a device, within a given session, as well as historical and profile infor and to help the user more effectively engage with local and/or 60 mation about the user. remote services. In various embodiments, the intelligent In addition, in various embodiments, context information automated assistant engages with the user in an integrated, derived from user interaction with a feature, operation, or conversational manner using natural language dialog, and application on a device can be used to streamline the opera invokes external services when appropriate to obtain infor tion of other features, operations, or applications on the mation or perform various actions. 65 device or on other devices. For example, the intelligent auto According to various embodiments of the present inven mated assistant can use the context of a phone call (such as the tion, the intelligent automated assistant integrates a variety of person called) to streamline the initiation of a text message US 9,318, 108 B2 3 4 (for example to determine that the text message should be sent to go (including but not limited to travel destinations, hotels to the same person, without the user having to explicitly and other places to stay, landmarks and other sites of interest, specify the recipient of the text message). The intelligent etc.); places to eat or drink (such as restaurants and bars), automated assistant of the present invention can thereby inter times and places to meet others, and any other source of pret instructions such as 'send him a text message', wherein 5 entertainment or social interaction which may be found on the the “him is interpreted according to context information Internet. Additionally, at least one intelligent automated derived from a current phone call, and/or from any feature, assistant system embodiment disclosed herein may be con operation, or application on the device. In various embodi figured or designed to include functionality for enabling the ments, the intelligent automated assistant takes into account operation of applications and services via natural language various types of available context data to determine which 10 dialog that may be otherwise provided by dedicated applica address book contact to use, which contact data to use, which telephone number to use for the contact, and the like, so that tions with graphical user interfaces including search (includ the user need not re-specify such information manually. ing location-based search); navigation (maps and directions); In various embodiments, the assistant can also take into database lookup (Such as finding businesses or people by account external events and respond accordingly, for 15 name or other properties); getting weather conditions and example, to initiate action, initiate communication with the forecasts, checking the price of market items or status of user, provide alerts, and/or modify previously initiated action financial transactions; monitoring traffic or the status of in view of the external events. If input is required from the flights; accessing and updating calendars and schedules; user, a conversational interface can again be used. managing reminders, alerts, tasks and projects; communicat In one embodiment, the system is based on sets of interre- 20 ing over email or other messaging platforms; and operating lated domains and tasks, and employs additional functionally devices locally or remotely (e.g., dialing telephones, control powered by external services with which the system can ling light and temperature, controlling home security devices, interact. In various embodiments, these external services playing music or video, etc.). Further, at least one intelligent include web-enabled services, as well as functionality related automated assistant system embodiment disclosed herein to the hardware device itself. For example, in an embodiment 25 may be configured or designed to include functionality for where the intelligent automated assistant is implemented on a identifying, generating, and/or providing personalized rec Smartphone, personal digital assistant, tablet computer, or ommendations for activities, products, services, Source of other device, the assistant can control many operations and entertainment, time management, or any other kind of rec functions of the device, such as to dial a telephone number, ommendation service that benefits from an interactive dialog send a text message, set reminders, add events to a calendar, 30 and the like. in natural language and automated access to data and Ser In various embodiments, the system of the present inven vices. tion can be implemented to provide assistance in any of a In various embodiments, the intelligent automated assis number of different domains. Examples include: tant of the present invention can control many features and Local Services (including location- and time-specific Ser- 35 operations of an electronic device. For example, the intelli vices Such as restaurants, movies, automated teller gent automated assistant can call services that interface with machines (ATMs), events, and places to meet); functionality and applications on a device via APIs or by other Personal and Social Memory Services (including action means, to perform functions and operations that might other items, notes, calendar events, shared links, and the like): wise be initiated using a conventional user interface on the E-commerce (including online purchases of items such as 40 device. Such functions and operations may include, for books, DVDs, music, and the like): example, setting an alarm, making a telephone call, sending a Travel Services (including flights, hotels, attractions, and text message or email message, adding a calendar event, and the like). the like. Such functions operations may be performed as One skilled in the art will recognize that the above list of add-on functions in the context of a conversational dialog domains is merely exemplary. In addition, the system of the 45 between a user and the assistant. Such functions and opera present invention can be implemented in any combination of tions can be specified by the user in the context of such a domains. dialog, or they may be automatically performed based on the In various embodiments, the intelligent automated assis context of the dialog. One skilled in the art will recognize that tant systems disclosed herein may be configured or designed the assistant can thereby be used as a control mechanism for to include functionality for automating the application of data 50 initiating and controlling various operations on the electronic and services available over the Internet to discover, find, device, which may be used as an alternative to conventional choose among, purchase, reserve, or order products and Ser mechanisms such as buttons or graphical user interfaces. vices. In addition to automating the process of using these data and services, at least one intelligent automated assistant BRIEF DESCRIPTION OF THE DRAWINGS system embodiment disclosed herein may also enable the 55 combined use of several sources of data and services at once. The accompanying drawings illustrate several embodi For example, it may combine information about products ments of the invention and, together with the description, from several review sites, check prices and availability from serve to explain the principles of the invention according to multiple distributors, and check their locations and time con the embodiments. One skilled in the art will recognize that the straints, and help a user find a personalized solution to their 60 particular embodiments illustrated in the drawings are merely problem. Additionally, at least one intelligent automated exemplary, and are not intended to limit the scope of the assistant system embodiment disclosed herein may be con present invention. figured or designed to include functionality for automating FIG. 1 is a block diagram depicting an example of one the use of data and services available over the Internet to embodiment of an intelligent automated assistant system. discover, investigate, select among, reserve, and otherwise 65 FIG. 2 illustrates an example of an interaction between a learn about things to do (including but not limited to movies, user and an intelligent automated assistant according to at events, performances, exhibits, shows and attractions); places least one embodiment. US 9,318, 108 B2 5 6 FIG. 3 is a block diagram depicting a computing device FIG. 37 is a flow diagram depicting an example of a pro Suitable for implementing at least a portion of an intelligent cedure for executing a service orchestration procedure automated assistant according to at least one embodiment. according to one embodiment. FIG. 4 is a block diagram depicting an architecture for FIG.38 is a flow diagram depicting an example of a service implementing at least a portion of an intelligent automated 5 invocation procedure according to one embodiment. assistant on a standalone computing system, according to at FIG. 39 is a flow diagram depicting an example of a mul least one embodiment. tiphase output procedure according to one embodiment. FIG. 5 is a block diagram depicting an architecture for FIGS. 40 and 41 are screen shots depicting examples of implementing at least a portion of an intelligent automated output processing according to one embodiment. assistant on a distributed computing network, according to at 10 least one embodiment. FIG. 42 is a flow diagram depicting an example of multi FIG. 6 is a block diagram depicting a system architecture modal output processing according to one embodiment. illustrating several different types of clients and modes of FIGS. 43A and 43B are screen shots depicting an example operation. of the use of short term personal memory component(s) to FIG. 7 is a block diagram depicting a client and a server, 15 maintain dialog context while changing location, according which communicate with each other to implement the present to one embodiment. invention according to one embodiment. FIGS. 44A through 44C are screen shots depicting an FIG. 8 is a block diagram depicting a fragment of an active example of the use of long term personal memory ontology according to one embodiment. component(s), according to one embodiment. FIG. 9 is a block diagram depicting an example of an FIG. 45 depicts an example of an abstract model for a alternative embodiment of an intelligent automated assistant constrained selection task. system. FIG. 46 depicts an example of a dialog flow model to help FIG. 10 is a flow diagram depicting a method of operation guide the user through a search process. for active input elicitation component(s) according to one FIG. 47 is a flow diagram depicting a method of con embodiment. 25 strained selection according to one embodiment. FIG. 11 is a flow diagram depicting a method for active typed-input elicitation according to one embodiment. DETAILED DESCRIPTION OF THE FIGS. 12 to 21 are screen shots illustrating some portions EMBODIMENTS of some of the procedures for active typed-input elicitation according to one embodiment. 30 Various techniques will now be described in detail with FIG. 22 is a flow diagram depicting a method for active reference to a few example embodiments thereof as illus input elicitation for voice or speech input according to one trated in the accompanying drawings. In the following embodiment. description, numerous specific details are set forth in order to FIG. 23 is a flow diagram depicting a method for active provide a thorough understanding of one or more aspects input elicitation for GUI-based input according to one 35 and/or features described or reference herein. It will be appar embodiment. ent, however, to one skilled in the art, that one or more aspects FIG. 24 is a flow diagram depicting a method for active and/or features described or reference herein may be prac input elicitation at the level of a dialog flow according to one ticed without some or all of these specific details. In other embodiment. instances, well known process steps and/or structures have FIG. 25 is a flow diagram depicting a method for active 40 not been described in detail in order to not obscure some of monitoring for relevant events according to one embodiment. the aspects and/or features described or reference herein. FIG. 26 is a flow diagram depicting a method for multimo One or more different inventions may be described in the dal active input elicitation according to one embodiment. present application. Further, for one or more of the FIG. 27 is a set of screen shots illustrating an example of invention(s) described herein, numerous embodiments may various types of functions, operations, actions, and/or other 45 be described in this patent application, and are presented for features which may be provided by domain models compo illustrative purposes only. The described embodiments are nent(s) and services orchestration according to one embodi not intended to be limiting in any sense. One or more of the ment. invention(s) may be widely applicable to numerous embodi FIG.28 is a flow diagram depicting an example of a method ments, as is readily apparent from the disclosure. These for natural language processing according to one embodi 50 embodiments are described in sufficient detail to enable those ment. skilled in the art to practice one or more of the invention(s), FIG. 29 is a screen shot illustrating natural language pro and it is to be understood that other embodiments may be cessing according to one embodiment. utilized and that structural, logical, Software, electrical and FIGS.30 and 31 are screen shots illustrating an example of other changes may be made without departing from the scope various types of functions, operations, actions, and/or other 55 of the one or more of the invention(s). Accordingly, those features which may be provided by dialog flow processor skilled in the art will recognize that the one or more of the component(s) according to one embodiment. invention(s) may be practiced with various modifications and FIG. 32 is a flow diagram depicting a method of operation alterations. Particular features of one or more of the for dialog flow processor component(s) according to one invention(s) may be described with reference to one or more embodiment. 60 particular embodiments or figures that form a part of the FIG.33 is a flow diagram depicting an automatic call and present disclosure, and in which are shown, by way of illus response procedure, according to one embodiment. tration, specific embodiments of one or more of the FIG.34 is a flow diagram depicting an example of task flow invention(s). It should be understood, however, that such for a constrained selection task according to one embodiment. features are not limited to usage in the one or more particular FIGS. 35 and 36 are screen shots illustrating an example of 65 embodiments or figures with reference to which they are the operation of constrained selection task according to one described. The present disclosure is neither a literal descrip embodiment. tion of all embodiments of one or more of the invention(s) nor US 9,318, 108 B2 7 8 a listing of features of one or more of the invention(s) that U.S. patent application Ser. No. 11/518.292 for “Method must be present in all embodiments. And Apparatus for Building an Intelligent Automated Headings of sections provided in this patent application Assistant, filed Sep. 8, 2006, the disclosure of which is and the title of this patent application are for convenience incorporated herein by reference; and only, and are not to be taken as limiting the disclosure in any 5 U.S. Provisional Patent Application Ser. No. 61/186,414 way. for "System and Method for Semantic Auto-Comple Devices that are in communication with each other need tion', filed Jun. 12, 2009, the disclosure of which is not be in continuous communication with each other, unless incorporated herein by reference. expressly specified otherwise. In addition, devices that are in Hardware Architecture communication with each other may communicate directly or 10 Generally, the intelligent automated assistant techniques indirectly through one or more intermediaries. disclosed herein may be implemented on hardware or a com A description of an embodiment with several components bination of software and hardware. For example, they may be in communication with each other does not imply that all Such implemented in an operating system kernel, in a separate user components are required. To the contrary, a variety of process, in a library package bound into network applica optional components are described to illustrate the wide vari 15 tions, on a specially constructed machine, or on a network ety of possible embodiments of one or more of the interface card. In a specific embodiment, the techniques dis invention(s). closed herein may be implemented in Software such as an Further, although process steps, method steps, algorithms operating system or in an application running on an operating or the like may be described in a sequential order. Such pro system. cesses, methods and algorithms may be configured to work in Software/hardware hybrid implementation(s) of at least alternate orders. In other words, any sequence or order of Some of the intelligent automated assistant embodiment(s) steps that may be described in this patent application does not, disclosed herein may be implemented on a programmable in and of itself, indicate a requirement that the steps be per machine selectively activated or reconfigured by a computer formed in that order. The steps of described processes may be program stored in memory. Such network devices may have performed in any order practical. Further, some steps may be 25 multiple network interfaces which may be configured or performed simultaneously despite being described or implied designed to utilize different types of network communication as occurring non-simultaneously (e.g., because one step is protocols. A general architecture for some of these machines described after the other step). Moreover, the illustration of a may appear from the descriptions disclosed herein. Accord process by its depiction in a drawing does not imply that the ing to specific embodiments, at least Some of the features illustrated process is exclusive of other variations and modi 30 and/or functionalities of the various intelligent automated fications thereto, does not imply that the illustrated process or assistant embodiments disclosed herein may be implemented any of its steps are necessary to one or more of the on one or more general-purpose network host machines such invention(s), and does not imply that the illustrated process is as an end-user computer system, computer, network server or preferred. server system, mobile computing device (e.g., personal digi When a single device or article is described, it will be 35 tal assistant, mobile phone, Smartphone, laptop, tablet com readily apparent that more than one device/article (whether or puter, or the like), consumer electronic device, music player, not they cooperate) may be used in place of a single device? or any other suitable electronic device, router, switch, or the article. Similarly, where more than one device or article is like, or any combination thereof. In at least Some embodi described (whether or not they cooperate), it will be readily ments, at least Some of the features and/or functionalities of apparent that a single device/article may be used in place of 40 the various intelligent automated assistant embodiments dis the more than one device or article. closed herein may be implemented in one or more virtualized The functionality and/or the features of a device may be computing environments (e.g., network computing clouds, or alternatively embodied by one or more other devices that are the like). not explicitly described as having such functionality/features. Referring now to FIG. 3, there is shown a block diagram Thus, other embodiments of one or more of the invention(s) 45 depicting a computing device 60 Suitable for implementing at need not include the device itself. least a portion of the intelligent automated assistant features Techniques and mechanisms described or reference herein and/or functionalities disclosed herein. Computing device 60 will sometimes be described in singular form for clarity. may be, for example, an end-user computer system, network However, it should be noted that particular embodiments server or server system, mobile computing device (e.g., per include multiple iterations of a technique or multiple instan 50 Sonal digital assistant, mobile phone, Smartphone, laptop, tiations of a mechanism unless noted otherwise. tablet computer, or the like), consumer electronic device, Although described within the context of intelligent auto music player, or any other suitable electronic device, or any mated assistant technology, it may be understood that the combination orportion thereof. Computing device 60 may be various aspects and techniques described herein (such as adapted to communicate with other computing devices. Such those associated with active ontologies, for example) may 55 as clients and/or servers, over a communications network also be deployed and/or applied in other fields of technology Such as the Internet, using known protocols for Such commu involving human and/or computerized interaction with Soft nication, whether wireless or wired. Wa. In one embodiment, computing device 60 includes central Other aspects relating to intelligent automated assistant processing unit (CPU) 62, interfaces 68, and a bus 67 (such as technology (e.g., which may be utilized by, provided by, 60 a peripheral component interconnect (PCI) bus). When acting and/or implemented at one or more intelligent automated under the control of appropriate software or firmware, CPU assistant system embodiments described herein) are dis 62 may be responsible for implementing specific functions closed in one or more of the following references: associated with the functions of a specifically configured U.S. Provisional Patent Application Ser. No. 61/295,774 computing device or machine. For example, in at least one for “Intelligent Automated Assistant, filed Jan. 18, 65 embodiment, a user's personal digital assistant (PDA) may be 2010, the disclosure of which is incorporated herein by configured or designed to function as an intelligent automated reference; assistant system utilizing CPU 62, memory 61, 65, and inter US 9,318, 108 B2 9 10 face(s) 68. In at least one embodiment, the CPU 62 may be purpose network operations and/or other information relating caused to perform one or more of the different types of intel to the functionality of the intelligent automated assistant tech ligent automated assistant functions and/or operations under niques described herein. The program instructions may con the control of software modules/components, which for trol the operation of an operating system and/or one or more example, may include an operating system and any appropri applications, for example. The memory or memories may ate applications software, drivers, and the like. also be configured to store data structures, keyword tax CPU 62 may include one or more processor(s) 63 such as, onomy information, advertisement information, user click for example, a processor from the Motorola or Intel family of and impression information, and/or other specific non-pro microprocessors or the MIPS family of microprocessors. In gram information described herein. Some embodiments, processor(s) 63 may include specially 10 Because Such information and program instructions may designed hardware (e.g., application-specific integrated cir be employed to implement the systems/methods described cuits (AS-ICs), electrically erasable programmable read-only herein, at least Some network device embodiments may memories (EEPROMs), field-programmable gate arrays (FP include nontransitory machine-readable storage media, GAS), and the like) for controlling the operations of comput which, for example, may be configured or designed to store ing device 60. In a specific embodiment, a memory 61 (Such 15 program instructions, state information, and the like for per as non-volatile random access memory (RAM) and/or read forming various operations described herein. Examples of only memory (ROM)) also forms part of CPU 62. However, Such nontransitory machine-readable storage media include, there are many different ways in which memory may be but are not limited to, magnetic media Such as hard disks, coupled to the system. Memory block 61 may be used for a floppy disks, and magnetic tape, optical media Such as CD variety of purposes such as, for example, caching and/or ROM disks; magneto-optical media Such as floptical disks, storing data, programming instructions, and the like. and hardware devices that are specially configured to store As used herein, the term “processor is not limited merely and perform program instructions, such as read-only memory to those integrated circuits referred to in the art as a processor, devices (ROM), flash memory, memristor memory, random but broadly refers to a microcontroller, a microcomputer, a access memory (RAM), and the like. Examples of program programmable logic controller, an application-specific inte 25 instructions include both machine code. Such as produced by grated circuit, and any other programmable circuit. a compiler, and files containing higher level code that may be In one embodiment, interfaces 68 are provided as interface executed by the computer using an interpreter. cards (sometimes referred to as “line cards”). Generally, they In one embodiment, the system of the present invention is control the sending and receiving of data packets over a implemented on a standalone computing system. Referring computing network and sometimes Support other peripherals 30 now to FIG. 4, there is shown a block diagram depicting an used with computing device 60. Among the interfaces that architecture for implementing at least a portion of an intelli may be provided are Ethernet interfaces, frame relay inter gent automated assistant on a standalone computing System, faces, cable interfaces, DSL interfaces, token ring interfaces, according to at least one embodiment. Computing device 60 and the like. In addition, various types of interfaces may be includes processor(s) 63 which run software for implement provided such as, for example, universal serial bus (USB), 35 ing intelligent automated assistant 1002. Input device 1206 Serial, Ethernet, Firewire, PCI, parallel, radio frequency can be of any type Suitable for receiving user input, including (RF), BluetoothTM, near-field communications (e.g., using for example a keyboard, touchscreen, microphone (for near-field magnetics), 802.11 (WiFi), frame relay, TCP/IP. example, for Voice input), mouse, touchpad, trackball, five ISDN, fast Ethernet interfaces, Gigabit Ethernet interfaces, way Switch, joystick, and/or any combination thereof. Output asynchronous transfer mode (ATM) interfaces, high-speed 40 device 1207 can be a screen, speaker, printer, and/or any serial interface (HSSI) interfaces, Point of Sale (POS) inter combination thereof. Memory 1210 can be random-access faces, fiber data distributed interfaces (FDDIs), and the like. memory having a structure and architecture as are known in Generally, such interfaces 68 may include ports appropriate the art, for use by processor(s) 63 in the course of running for communication with the appropriate media. In some Software. Storage device 1208 can be any magnetic, optical, cases, they may also include an independent processor and, in 45 and/or electrical storage device for storage of data in digital Some instances, Volatile and/or non-volatile memory (e.g., form; examples include flash memory, magnetic hard drive, RAM). CD-ROM, and/or the like. Although the system shown in FIG. 3 illustrates one spe In another embodiment, the system of the present invention cific architecture for a computing device 60 for implementing is implemented on a distributed computing network, Such as the techniques of the invention described herein, it is by no 50 one having any number of clients and/or servers. Referring means the only device architecture on which at least a portion now to FIG. 5, there is shown a block diagram depicting an of the features and techniques described herein may be imple architecture for implementing at least a portion of an intelli mented. For example, architectures having one or any number gent automated assistant on a distributed computing network, of processors 63 can be used, and such processors 63 can be according to at least one embodiment. present in a single device or distributed among any number of 55 In the arrangement shown in FIG. 5, any number of clients devices. In one embodiment, a single processor 63 handles 1304 are provided; each client 1304 may run software for communications as well as routing computations. In various implementing client-side portions of the present invention. In embodiments, different types of intelligent automated assis addition, any number of servers 1340 can be provided for tant features and/or functionalities may be implemented in an handling requests received from clients 1304. Clients 1304 intelligent automated assistant system which includes a client 60 and servers 1340 can communicate with one another via device (such as a personal digital assistant or Smartphone electronic network 1361, such as the Internet. Network 1361 running client Software) and server System(s) (Such as a may be implemented using any known network protocols, server system described in more detail below). including for example wired and/or wireless protocols. Regardless of network device configuration, the system of In addition, in one embodiment, servers 1340 can call the present invention may employ one or more memories or 65 external services 1360 when needed to obtain additional memory modules (such as, for example, memory block 65) information or refer to store data concerning previous inter configured to store data, program instructions for the general actions with particular users. Communications with external