US00835,2277B2 (12) United States Patent (10) Patent No.: US 8,352.277 B2 Bennett (45) Date of Patent: Jan. 8, 2013 (54) METHOD OF INTERACTING THROUGH 37.8 A 3. rittOnet et hal. SPEECH WITH A WEB-CONNECTED 4,852,170 A 7, 1989 Bordeaux SERVER 4,868,750 A 9/1989 Kucera et al. 4.914,590 A 4, 1990 Loatman et al. (75) Inventor: Ian M. Bennett, Palo Alto, CA (US) 4.937,870 A 6/1990 Bossemeyer, Jr. 4.956,865 A 9/1990 Lennig et al. (73) Assignee: Phoenix Solutions, Inc., Palo Alto, CA 4,991,2174,991,094. A 2f19912/1991 GarrettFagan et et al. al. (US) 5,036,539 A 7/1991 Wrench et al. 5,068,789 A 1 1/1991 Van Vliembergen (*) Notice: Subject to any disclaimer, the term of this 5,146,405 A 9, 1992 Church patent is extended or adjusted under 35 3.5% A 19 E. OR O. ca.tal U.S.C. 154(b) by 1209 days. 5,265,014 A 1 1/1993 Haddock et al. 5,278,980 A 1/1994 Pedersen et al. (21) Appl. No.: 11/733,129 5,293,584. A 3/1994 Brown et al. (22) Filed: Apr. 9, 2007 (Continued)Continued FOREIGN PATENT DOCUMENTS (65) Prior Publication Data EP 1094388 4/2001 US 2007/O1857 17 A1 Aug. 9, 2007 (Continued) Related U.S. Application Data OTHER PUBLICATIONS (60) Division of application No. 1 1/419,736, filed on May 21st Century Eloquence, Inc. (Archived Internet advertisement (C) 22, 2006, now Pat. No. 7,203,646, which is a 1997-1998), 9 pages. continuation of application No. 09/439.174, filed on Nov. 12, 1999, now Pat. No. 7,050,977. (Continued) (51) Int. Cl. Primary Examiner — Martin Lerner GOL 5/00 (2006.01) (74) Attorney, Agent, or Firm — David Lewis GIOL I5/22 (2006.01) 57 ABSTRACT (52) U.S. Cl. ...................... 704/270.1; 704/251; 704/270 (57) (58) Field of Classification Search .................. 704/255, A speech recognition engine is configured to permit partial or 704/257, 258, 270, 270.1, 275,231, 251, full recognition of an utterance at a client device and/or a 704/252; 707/3, 6 server device. This allows for support for client devices hav See application file for complete search history. ing differing computing capabilities. Natural language opera tions can also be supported as desired. A usercanthus interact (56) References Cited with content also accessible at a web page and select items of interest using speech as a mode of input. Dynamic grammars U.S. PATENT DOCUMENTS can assist in the recognition operations to improve speed and 4473,904 A 9, 1984 Suehiro et al. comprehension. 4,521,907 A 6, 1985 Amir et al. 4,587,670 A 5, 1986 Levinson et al. 20 Claims, 31 Drawing Sheets SRE: Server-side 182 w CLIENT-SIDE a SERWR-site * US 8,352.277 B2 Page 2 U.S. PATENT DOCUMENTS 5,978,756 11, 1999 Walker et al. 5,983, 190 11, 1999 Trower, II et al. 5,317,507 5, 1994 Gallant 5,987.404 11, 1999 Della Pietra et al. 5,371,901 12, 1994 Reed et al. 5,987,410 11, 1999 Kellner et al. 5,384,892 1, 1995 Strong 5,995,918 11, 1999 Kendall et al. 5,454,106 9, 1995 Burns et al. 5,995,928 11, 1999 Nguyen et al. 5,475,792 12, 1995 Stanford et al. 6,006,221 12, 1999 Liddy et al. 5,500,920 3, 1996 Kupiec 6,009,387 12, 1999 Ramaswamy et al. 5,509, 104 4, 1996 Lee et al. 6,018,736 1, 2000 Gilai et al. 5,513.298 4, 1996 Stanford et al. 6,021,384 2, 2000 Gorin et al. 5,519,608 5, 1996 Kupiec 6,023,697 2, 2000 Bates et al. 5,524,169 6, 1996 Cohen et al. 6,029, 124 2, 2000 Gillick et al. 5,540,589 T. 1996 Waters 6,032,111 2, 2000 Mohri 5,553,119 9, 1996 McAllister et al. 6,035,275 3, 2000 Brode et al. 5,602,963 2, 1997 Bissonnette et al. 6,038,535 3, 2000 Campbell 5,615,296 3, 1997 Stanford et al. 6,044.266 3, 2000 Kato 5,625,748 4, 1997 McDonough et al. 6,044,337 3, 2000 Gorin et al. 5,625,814 4, 1997 Luciw 6,044,347 3, 2000 Abella et al. 5,630,121 5, 1997 Braden-Harder et al. 6,052,439 4, 2000 GerSzberg et al. 5,652,789 7, 1997 Miner et al. 6,052,656 4, 2000 Suda et al. 5,652,897 7, 1997 Linebarger et al. 6,061,646 5/2000 Martino et al. 5,668,854 9, 1997 Minakami et al. 6,076,088 6, 2000 Paik et al. 5,675.707 10, 1997 Gorin et al. 6,078,886 6, 2000 Dragosh et al. 5,675,788 10, 1997 Husick et al. 6,078,914 6, 2000 Redfern 5,675,819 10, 1997 Schuetze 6,081,774 6, 2000 De Hita et al. 5,680,511 10, 1997 Baker et al. 6,085,160 T/2000 D'Hoore et al. 5,680,628 10, 1997 Carus et al. 6,088,692 T/2000 Driscoll 5,689,617 11, 1997 Pallakoffet al. 6,094,635 T/2000 Scholz et al. 5,694,592 12, 1997 Driscoll 6,101,472 8, 2000 Giangarra et al. 5,696,962 12, 1997 Kupiec 6,105,023 8, 2000 Callan 5,699.456 12, 1997 Brown et al. 6,112, 176 8, 2000 Goldenthal et al. 5,703,935 12, 1997 Raissyan et al. 6,119,087 9, 2000 Kuhn et al. 5,719,997 2, 1998 Brown et al. 6,122,613 9, 2000 Baker 5,721,938 2, 1998 Stuckey 6,125,284 9, 2000 Moore et al. 5,727,950 3, 1998 Cook et al. 6,125,341 9, 2000 Raud et al. 5,730,603 3, 1998 Harless 6,131,085 10, 2000 Rossides 5,737,485 4, 1998 Flanagan et al. 6,138,087 10, 2000 Budzinski 5,748,841 5, 1998 Morin et al. 6,138,088 10, 2000 Goeser 5,748,974 5, 1998 Johnson 6,138,089 10, 2000 Guberman 5,751,907 5, 1998 Moebius et al. 6,138,091 10, 2000 Haataja et al. 5,758,023 5, 1998 Bordeaux 6,138,100 10, 2000 Dutton et al. 5,758,322 5, 1998 Rongley 6,141,640 10, 2000 Moo 5,765,130 6, 1998 Nguyen 6,144,848 11, 2000 Walsh et al. 5,774,841 6, 1998 Salazar et al. 6,144.938 11, 2000 Surace et al. 5,774,859 6, 1998 Houser et al. 6,157,705 12, 2000 Perrone 5,794,050 8, 1998 Dahlgren et al. 6,157,909 12, 2000 Mauuary et al. 5,794, 193 8, 1998 Gorin 6,157,912 12, 2000 Kneser et al. 5,797,123 8, 1998 Chou et al. 6,173,260 1, 2001 Slaney 5,802,251 9, 1998 Cohen et al. 6,173,261 1, 2001 Arai et al. 5,802.256 9, 1998 Heckerman et al. 6,173,266 1, 2001 Marx et al. 5,802.526 9, 1998 Fawcett et al. 6,173,279 1, 2001 Levin et al. 5,819,220 10, 1998 Sarukkai et al. 6,175,634 1, 2001 Graumann 5,826,227 10, 1998 Jayant 6,178.404 1, 2001 Hambleton et al. 5,836,771 11, 1998 Ho et al. 6,182,038 1, 2001 Balakrishnan et al. 5,838,683 11, 1998 Corley et al. 6,182,068 1, 2001 Culliss 5,839,106 11, 1998 Bellegarda 6,185,535 2, 2001 Hedin et al. 5,842,196 11, 1998 Agarwal et al. 6,188,976 2, 2001 Ramaswamy et al. 5,855,002 12, 1998 Armstrong 6, 192,110 2, 2001 Abella et al. 5,860,063 1, 1999 Gorin et al. 6, 192,338 2, 2001 HaSzto et al. 5,865,626 2, 1999 Beattie et al. 6, 195,636 2, 2001 Crupi et al. 5,867,817 2, 1999 Catallo et al. 6,199,041 3, 2001 Liu et al. 5,870,706 2, 1999 Alshawi 6,216,013 4, 2001 Moore et al. 5,873,062 2, 1999 Hansen et al. 6,219,643 4, 2001 Cohen et al. 5,878.406 3, 1999 Noyes 6,226,610 5/2001 Keiller et al. 5,878.423 3, 1999 Anderson et al. 6,233.559 5/2001 Balakrishnan 5,884,302 3, 1999 Ho 6,243,679 6, 2001 Mohri et al. 5,895,466 4, 1999 Goldberg et al. 6,246,977 6, 2001 Messerly et al. 5,897,616 4, 1999 Kanevsky et al. 6,246,981 6, 2001 Papineni et al. 5,905,773 5, 1999 Wong 6,246,986 6, 2001 Ammicht et al. 5,913, 193 6, 1999 Huang et al. 6, 2001 Polcyn 5,915,236 6, 1999 Gould et al. 6,246,989 5,933,822 8, 1999 Braden-Harder et al. 6.256,607 T/2001 Digalakis et al. 5,934,910 8, 1999 Ho et al. 6,260,008 T/2001 Sanfilippo 5,953,701 9, 1999 Neti et al. 6,269,153 T/2001 Carpenter et al. 5,956,675 9, 1999 Setlur et al. 6,269,336 T/2001 Ladd et al. 5,956,683 9, 1999 Jacobs et al.