USOO911 6890B2 (12) United States Patent (10) Patent No.: US 9,116,890 B2 King et al. (45) Date of Patent: *Aug. 25, 2015 (54) TRIGGERING ACTIONS IN RESPONSE TO (52) U.S. Cl. OPTICALLY ORACOUSTICALLY CPC .......... G06F 17/30011 (2013.01); G06K 9/344 CAPTURING KEYWORDS FROMA (2013.01) RENDERED DOCUMENT (58) Field of Classification Search CPC .......... G06F 17/30011; G06F 17/3002; G06F (71) Applicant: Google Inc., Mountain View, CA (US) 17/30026; G06F 17/30029; G06F 17/30047; H04L 49/00; H04L 49/3009; H04L 49/356; (72) Inventors: Martin T. King, Vashon Island, WA H04L 49/357; H04L 49/50: H04L 49/55; (US); Dale L. Grover, Ann Arbor, MI H04L 49/70; H04L 63/0227 (US); Clifford A. Kushler, Lynnwood, See application file for complete search history. WA (US); James Q. Stafford-Fraser, (56) References Cited Cambridge (GB) U.S. PATENT DOCUMENTS (73) Assignee: Google Inc., Mountain View, CA (US) 3,899,687 A 8, 1975 Jones (*) Notice: Subject to any disclaimer, the term of this 4,052,0583,917,317 A 10/197711/1975 HintzRyan patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days. (Continued) This patent is Subject to a terminal dis- FOREIGN PATENT DOCUMENTS claimer. EP O424803 5, 1991 (21) Appl. No.: 14/302,023 EP O544434 6, 1993 (Continued) (22) Filed: Jun. 11, 2014 OTHER PUBLICATIONS (65) Prior Publication Data European Search Report for EP Application No. 05733819 dated Mar. 31, 2009. US 2014/02943O2 A1 Oct. 2, 2014 (Continued) Primary Examiner — Jose Couso Related U.S. Application Data (74) Attorney, Agent, or Firm — Fish & Richardson P.C. (60) Continuation of application No. 13/614,770, filed on (57) ABSTRACT Sep. 13, 2012, now Pat. No. 8,781,228, which is a A system for processing text captured from rendered docu continuation of application No. 13/031,316, filed on ments is described. The system receives a sequence of one or Feb. 21, 2011, now Pat. No. 8,447,111, which is a more words optically or acoustically captured from a ren dered document by a user. The system identifies among words (Continued) of the sequence a word with which an action has been asso (51) Int. Cl ciated. The system then performs the associated action with G06K 9/34 (2006.01) respect to the user. G06F 7/30 (2006.01) 20 Claims, 8 Drawing Sheets 38arch & cortext arlaysis US 9,116,890 B2 Page 2 Related U.S. Application Data Aug. 27, 2004, provisional application No. 60/605, 105, filed on Aug. 27, 2004, provisional application continuation of application No. 12/538,731, filed on No. 60/604,103, filed on Aug. 23, 2004, provisional Aug. 10, 2009, now Pat. No. 7,894,670, which is a application No. 60/604,098, filed on Aug. 23, 2004, continuation of application No. 1 1/097,103, filed on provisional application No. 60/604,100, filed on Aug. Apr. 1, 2005, now Pat. No. 7,596.269, which is a con 23, 2004, provisional application No. 60/604,102, tinuation-in-part of application No. 11/004,637, filed filed on Aug. 23, 2004, provisional application No. on Dec. 3, 2004, now Pat. No. 7,707,039, and a divi 60/603,498, filed on Aug. 20, 2004, provisional appli sion of application No. 60/653,899, filed on Feb. 16, cation No. 60/603,358, filed on Aug. 20, 2004, provi 2005. sional application No. 60/603,466, filed on Aug. 19, (60) Provisional application No. 60/657,309, filed on Feb. 2004, provisional application No. 60/603,082, filed on 28, 2005, provisional application No. 60/655,697, Aug. 19, 2004, provisional application No. 60/603, filed on Feb. 22, 2005, provisional application No. 081, filed on Aug. 19, 2004, provisional application 60/655.279, filed on Feb. 22, 2005, provisional appli No. 60/602.956, filed on Aug. 18, 2004, provisional cation No. 60/655,281, filed on Feb. 22, 2005, provi application No. 60/602.925, filed on Aug. 18, 2004, sional application No. 60/655,280, filed on Feb. 22, provisional application No. 60/602,947, filed on Aug. 2005, provisional application No. 60/655,987, filed on 18, 2004, provisional application No. 60/602,897, Feb. 22, 2005, provisional application No. 60/654,196, filed on Aug. 18, 2004, provisional application No. filed on Feb. 18, 2005, provisional application No. 60/602.896, filed on Aug. 18, 2004, provisional appli 60/654,326, filed on Feb. 18, 2005, provisional appli cation No. 60/602,930, filed on Aug. 18, 2004, provi cation No. 60/654.368, filed on Feb. 18, 2005, provi sional application No. 60/602,898, filed on Aug. 18, sional application No. 60/654,379, filed on Feb. 17, 2004, provisional application No. 60/598.821, filed on 2005, provisional application No. 60/653,669, filed on Aug. 2, 2004, provisional application No. 60/589.203, Feb. 16, 2005, provisional application No. 60/653,663, filed on Jul. 19, 2004, provisional application No. filed on Feb. 16, 2005, provisional application No. 60/653,679, filed on Feb. 16, 2005, provisional appli 60/589.201, filed on Jul. 19, 2004, provisional appli cation No. 60/653,847, filed on Feb. 16, 2005, provi cation No. 60/589.202, filed on Jul. 19, 2004, provi sional application No. 60/653,372, filed on Feb. 15, sional application No. 60/.571,715, filed on May 17, 2005, provisional application No. 60/648,746, filed on 2004, provisional application No. 60/571,381, filed on Jan. 31, 2005, provisional application No. 60/647,684, May 14, 2004, provisional application No. 60/.571, filed on Jan. 26, 2005, provisional application No. 560, filed on May 14, 2004, provisional application 60/634,627, filed on Dec. 9, 2004, provisional appli No. 60/566,667, filed on Apr. 30, 2004, provisional cation No. 60/634,739, filed on Dec. 9, 2004, provi application No. 60/564,688, filed on Apr. 23, 2004, sional application No. 60/633,486, filed on Dec. 6, provisional application No. 60/564,846, filed on Apr. 2004, provisional application No. 60/633,678, filed on 23, 2004, provisional application No. 60/563,520, Dec. 6, 2004, provisional application No. 60/633,452, filed on Apr. 19, 2004, provisional application No. filed on Dec. 6, 2004, provisional application No. 60/563,485, filed on Apr. 19, 2004, provisional appli 60/633,453, filed on Dec. 6, 2004, provisional appli cation No. 60/561,768, filed on Apr. 12, 2004, provi cation No. 60/622,906, filed on Oct. 28, 2004, provi sional application No. 60/559,766, filed on Apr. 6, sional application No. 60/615,378, filed on Oct. 1, 2004, provisional application No. 60/559,125, filed on 2004, provisional application No. 60/615, 112, filed on Apr. 2, 2004, provisional application No. 60/558,909. Oct. 1, 2004, provisional application No. 60/615,538, filed on Apr. 2, 2004, provisional application No. filed on Oct. 1, 2004, provisional application No. 60/559,033, filed on Apr. 2, 2004, provisional applica 60/613.243, filed on Sep. 27, 2004, provisional appli tion No. 60/559,127, filed on Apr. 2, 2004, provisional cation No. 60/613,628, filed on Sep. 27, 2004, provi application No. 60/559,087, filed on Apr. 2, 2004, pro sional application No. 60/613,632, filed on Sep. 27. visional application No. 60/559,131, filed on Apr. 2, 2004, provisional application No. 60/613,589, filed on 2004, provisional application No. 60/559,226, filed on Sep. 27, 2004, provisional application No. 60/613,242, Apr. 1, 2004, provisional application No. 60/559,893, filed on Sep. 27, 2004, provisional application No. filed on Apr. 1, 2004, provisional application No. 60/613,602, filed on Sep. 27, 2004, provisional appli 60/558,968, filed on Apr. 1, 2004, provisional applica cation No. 60/613,340, filed on Sep. 27, 2004, provi tion No. 60/558,867, filed on Apr. 1, 2004, provisional sional application No. 60/613,634, filed on Sep. 27. application No. 60/559.278, filed on Apr. 1, 2004, pro 2004, provisional application No. 60/613,461, filed on visional application No. 60/559.279, filed on Apr. 1, Sep. 27, 2004, provisional application No. 60/613,455, 2004, provisional application No. 60/559,265, filed on filed on Sep. 27, 2004, provisional application No. Apr. 1, 2004, provisional application No. 60/559,277. 60/613,460, filed on Sep. 27, 2004, provisional appli filed on Apr. 1, 2004, provisional application No. cation No. 60/613,400, filed on Sep. 27, 2004, provi 60/558.969, filed on Apr. 1, 2004, provisional applica sional application No. 60/613,456, filed on Sep. 27. tion No. 60/558,892, filed on Apr. 1, 2004, provisional 2004, provisional application No. 60/613,341, filed on application No. 60/558,760, filed on Apr. 1, 2004, pro Sep. 27, 2004, provisional application No. 60/613,361, visional application No. 60/558,717, filed on Apr. 1, filed on Sep. 27, 2004, provisional application No. 2004, provisional application No. 60/558,499, filed on 60/613.454, filed on Sep. 27, 2004, provisional appli Apr. 1, 2004, provisional application No. 60/558,370, cation No. 60/613,339, filed on Sep. 27, 2004, provi filed on Apr. 1, 2004, provisional application No. sional application No. 60/613,633, filed on Sep. 27. 60/558.789, filed on Apr. 1, 2004, provisional applica 2004, provisional application No.