US008761513B1 (12) United States Patent (10) Patent No.: US 8,761,513 B1 Rogowski et al. (45) Date of Patent: Jun. 24, 2014 (54) SYSTEMS AND METHODS FOR DISPLAYING (56) References Cited FOREIGN CHARACTER SETS AND THEIR |U.S. PATENT DOCUMENTS TRANSLATIONS IN REAL TIME ON -- - RESOURCE-CONSTRAINED MOBILE 5,875,421 A 2/1999 Takeuchi DEVICES D453,766 S 2/2002 Grandcolas et al. (71) Applicants: Ryan Leon Rogowski, Naperville, IL (Continued) º FOREIGN PATENT DOCUMENTS EP 2587389 A1 5/2013 (72) Inventors: Ryan Leon Rogowski, Naperville, IL WO 2013040 107 A1 3/2013 (US); Huan-Yu Wu, Taipei (TW); Kevin WO 2013,134090 A1 9/2013 Anthony Clark, Warwick, RI (US) WO 201400 1937 A1 1/2014 OTHER PUBLICATIONS (73) Assignee: Translate Abroad, Inc., Providence, RI (US) Quest Visual Inc, “Word Lens,” Quest Visual website, available at http://questvisual.com/us/Accessed on Jan. 11, 2014. (*) Notice: Subject to any disclaimer, the term of this Continued patent is extended or adjusted under 35 (Continued) U.S.C. 154(b) by 0 days. Primary Examiner – Ruiping Li - (74) Attorney, Agent, or Firm – American Patent Agency (21) Appl. No.: 14/207,155 PC; Daniar Hussain; Karl Dresdner (22) Filed: Mar 12, 2014 (57) ABSTRACT The present invention is related to systems and methods for Related U.S. Application Data translating language text on a mobile camera device offline e K_f e without access to the Internet. More specifically, the present (60) Provisional application No. 61/791,584, filed on Mar. invention relates to systems and methods for displaying text 15, 2013. of a first language and a translation of the first language text into a second language text which is displayed in real time in (51) Int. Cl. augmented reality on the mobile device. The processing can G{}6K 9/00 (2006.01) use a single line or a multiline algorithm designed with a G06F H 7/28 (2006.01) plurality of processing innovations to insure accurate real (52) U.S. CI. time translations without motion jitter. The invention may be CPC .................................... Goof 17289 (2013.01) used to help travelers in a foreign country with difficulties in USPC 382/181: 382/182: 382/135: 382/187 reading and understanding text written in the local language 58) Field foi - - - - - .ficati s h s s of that country. The present invention may be utilized with (58) º O º º . º 9/4638: G06K 9/18: wearable computers or glasses, producing seamless aug mented reality foreign language translations. Some embodi G06K 9/222; G07D 7/20: G07D 7/12: H04M ments are particularly useful in translations from Asian lan 1/72522; HO4M 1/72527 guages to English. USPC .................................. 382/181, 182, 135, 187 See application file for complete search history. 30 Claims, 22 Drawing Sheets iTsiºr single T. | LINEALGORTHM #3-100 YES CROP IMAGE FROM | BOUNDING BOX i PRE-PROCESSING ;CHARACTER DETECTION) REFERENCE (ŠEE FIG. 2A) ---------------sºccursosº.YES 22 TEXT S – 12 #CHARACTERl SEGMENT RECOGM?THON l |-------------------- y 148 US 8,761,513 B1 Page 2 (56) References Cited Go Food Apps, “Escargo.” available at http://www.gofoodapps.com/ Accessed on Jan. 11, 2014. |U.S. PATENT DOCUMENTS Purdue University App Translator, “Purdue University App Transla tor.” Purdue University App Translator website, available at http:// D479,531 S 9/2003 Yanagida www.purdue.edu/newsroom/research/2011/ D486,499 S 2/2004 Hayashi et al. D523,440 S 6/2006 Hernandez et al. 110908BoutinMenutranslate.html Accessed on Jan. 11, 2014. D541,291 S 4/2007 Zhou et al. Clear Menu, “Clear Menu.” Clear Menu website, available at http:// D553,140 S 10/2007 Noviello et al. www.clear2all.com/site/pages/57 Accessed on Jan. 11, 2014. D601,154 S 9/2009 Pelczarski et al. Babelphone, “Babelphone,” Babelphone website, available at http:// 8,386,231 B2 2/2013 Lebeau et al. appworld.blackberry.com/webstore/content/3892/ D678,894 S 3/2013 Kanalakis et al. '?countrycode=US&lang=en Accessed on Jan. 11, 2014. D680,109 S 4/2013 Akana et al. Lonely Planet Phrase Books, “Lonely Planet Phrase Books.” Lonely D681,657 S 5/2013 Deng Planet Phrase Books website, available at http://www.lonelyplanet. D682,854 S 5/2013 Kanalakis et al. com/apps-and-ebooks/Accessed on Jan. 11, 2014. 8,484,218 B2 7/2013 Raghunath iLingual, “illingual,” iDingual website, available at https://itunes. D687,846 S 8/2013 Jou D694,258 S 11/2013 Lee et al. apple.com/us/app/ilingual-french/id331907531?mt=8 Accessed on D697.077 S 1/2014 Rodenhouse et al. Jan. 11, 2014. D697,078 S 1/2014 Carlin et al. World Nomads Language Guides, “World Nomads Language 2007/0050183 A1 3/2007 Kao et al. Guides.” World Nomads Language Guides website, available at 2010/0331043 A1* 12/2010 Chapman et al. .......... 455/556.1 http://journals.worldnomads.com/language-guides/ Accessed on 2011/0090253 A1 * 4/2011 Good ............................ 345/633 Jan. 11, 2014. 2012/0179448 A1 7/2012 Gupta et al. Speakerlike iPhone Translator, “SpeakLike iPhone Photo Transla 2012/0323707 A1 12/2012 Urban tor.” SpeakLike iPhone Photo Translator website, available at http:// 2012/0330642 A1 12/2012 Kalb et al. www.speaklike.com/products/iphone-photo-translator/Accessed on 2012/0330643 A1* 12/2012 Frei et al. .......................... 704/2 2013/0030789 A1 1/2013 Dalce Jan. 11, 2014. 2013/0041647 A1 2/2013 Ramerth et al. Translation Pronounciation, “Translation pronounciation.” Transla 2013/0103383 A1 * 4/2013 Du et al. ........................... 704/3 tion pronounciation website, available https://itunes.apple.com/us/ 2013/021 1814 A1 8/2013 Derks et al. app/60+-languages-translation+voice+pronunciation--ocr/ 2013/0226557 A1 8/2013 Uszkoreit et al. id345780733?mt=8 Accessed on Jan. 11, 2014. 2013/0234945 A1 9/2013 Goktekin Wordtrans, “Wordtrans.” Wordtrans website, available at http:// 2013/0238336 A1 9/2013 Sung et al. wordtrans.com/Accessed on Jan. 11, 2014. 2013/0253901 A1 9/2013 Krack et al. Camdictionary, “Cambictionary App,” available at https://play. 2013/0253903 A1 9/2013 Stephen et al. google.com/store/apps/details?id=com.intsig.camdict&hl=en 2013/0293577 A1 1 1/2013 Perez et al. Accessed on Jan. 11, 2014. 2014/0006006 A1 1/2014 Christ Photo Translator, “Photo Translator App,” available at https://itunes. 2014/0006007 A1 1/2014 Sumita et al. apple.com/us/app/photo-translator-free/id478334807?mt=8 OTHER PUBLICATIONS Accessed on Jan. 11, 2014. Translation Proununciation, “Translation Pronunciation App,” avail Pleco Software Inc, “Pleco Chinese Dictionary.” Pleco software able af https://itunes.apple.com/us/app/60+-languages website, available at https://www.pleco.com/ Accessed on Jan. 11, translation+voice+pronunciation+ocr/id345780733?mt=8 Accessed 2014. on Jan. 11, 2014. AT&T, “AT&T Translator.” AT&T website, available at http://www. Google Glass, “Google Glass Translation App,” available at http:// att.com/gen/press-room?pid=22626 Accessed on Jan. 11, 2014. www.mediabistro.com/appmewser/word-lens-brings-real-time Babelshot, “Babelshot Photo Translator App,” available https:// translation-app-to-google-glass bºº?8 Accessed on Jan. 11, 2014. itunes.apple.com/us/app/babelshot-photo-translator/ id334194705?mt=8 Accessed on Jan. 11, 2014. * cited by examiner U.S. Patent Jun. 24, 2014 Sheet 1 of 22 US 8,761,513 B1 Fig. 1A 150 START SINGLE LINEA_GORITHM ~... 100 ºvers |S NO FOCUSED2 2^ {DETERM #NES S No |MULTELINEALGORITHM SINGLELINE2. (SEE FIG 3) - CROP MAGE FROM BOUND}NG BOX \,...} 08 PRE-PROCESSING (CHARACTER L-110 DETECTION) REFERENCE (SEE FIG. 2A) 144 YES > TEXT . ... 112 PRECURSORT 2. (No --- 2-116 SHOW “TRY TO CHARACTER SEGMENT ZOOM IN RECOGNITION OR GET (SEE FIG 2B) CLOSER” 2118 AVERAGE CHARACTER ~SIZE SMALL is NO U.S. Patent Jun. 24, 2014 Sheet 2 of 22 US 8,761,513 B1 | Fig. 1B 122 FILTER OUT A NON-CHINESE CHARACTERS SHOW “HMAGE 1...N -124 UNCLEAR” YES Bl_ANK ºf OR “USE ~, STRENGT’ Fi_ASHLHGHT” ^. \-126 NO VARANT -130 CONVERSHON 2–132 TRANSLATION ...” CHECK IF STAYING L-134 AS SAME TEXT 2- 136 SHOW PREVIOUS RESULT IF TEXT {S STAYING AND PREVIOUS RESULT HAS A BETTER TRANSLATION SCORE, OTHERWISE, SHOW CURRENT RESULT U.S. Patent Jun. 24, 2014 Sheet 3 of 22 US 8,761,513 B1 Fig. 2A 200 START PRE-PROCESS|NG _-202 UPSAMPLENG THE CROPPED 2-206 GRAY |MAGE TO A FIXED SIZE DECIDE THRESHOLD TYPE IN 2-208 BINARIZATION: ADAPTIVE B?NARIZATION CONNECTED COMPONENT 2–210 ANALYSIS (CCA) DENO{SE END _-214 (GO TO 112 FIG 1A) U.S. Patent Jun. 24, 2014 Sheet 4 of 22 US 8,761,513 B1 Fig. 2B 200 216 - START CHARACTER SEGMENT RECOGNITION 218 - STORE TEXT PRECURSOR NFORMATION 220 -s, VERTICAL MERGE 222 --> REMOVE TEXT PRECURSORS OUTS?DE REGION EXTENDED FROM CENTER HOREZONTAL L; NE 224 – SORT TEXT PRECURSORS |N LEFT TO RIGHT ORDER END CHARACTER SEGMENT AVERAGE CHARACTER RECOGNITION S{ZE SMALL (GO TO 120) 230 - HORIZONTAL MERGE WITH RECOGNITION FEEDBACK (SEE FIG. 2C) ~ END (GO TO 122) U.S. Patent Jun. 24, 2014 Sheet 5 of 22 US 8,761,513 B1 START B?NARY _2~234 CHARACTER RECOGNITION SET REGION OF 2-238 |NTEREST ON THE _2^ B|NARY |MAGE ----_-240 NORMALIZATION FEATURE 2- 242 EXTRACTION D?MENSIONALTY 2- 244 REDUCTION - 246 CLASS|F}CATHON W?TH C# USTER?NG END BINARY CHARACTER … 248 RECOGNITION (RETURN 2’ RESULTS TO 230) U.S. Patent Jun. 24, 2014 Sheet 6 of 22 US 8,761,513 B1 Fig. 3A 300 starTMULTILINETL 302 OVERALL FLOWCHART SUB-SAMPLE L-304 #MAGE SAVE THE YES HMAGE - PAUSE PRESSED?. 2-312 DO TRACKING ON RESIZED PREV. MAGE AND : CURRENT IMAGE 2- 314 EXAMINE VECTOR RESULTS 2 316 VECTORS >< CONSISTENT2 º 348 AND X, Y ACCUMULATED SHIFT DIDN'T CHANGE TOO MUCH SINCE .