United States Patent (10) Patent No.: US 9,519,641 B2 Solntseva Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USOO9519641B2 (12) United States Patent (10) Patent No.: US 9,519,641 B2 Solntseva et al. (45) Date of Patent: Dec. 13, 2016 (54) PHOTOGRAPHY RECOGNITION 6,049,328 A 4/2000 Vanderheiden TRANSLATON 6,115,482 A * 9/2000 Sears et al. ................... 382,114 6,539,116 B2 * 3/2003 Takaoka ................... GO6K900 (71) Applicant: ABBYY Software Ltd., Nicosia (CY) 382, 176 6,937,974 B1* 8/2005 d’Agostini ........................ TO4/2 (72) Inventors: Ekaterina Solntseva, Moscow (RU); 7,254,531 B2* 8/2007 Nir .......................... G10. Konstantin Tarachyov, Moscow (RU) 7,499.588 B2 * 3/2009 Jacobs et al. ................. 382, 182 7,505,627 B2 * 3/2009 Chang et al. ...... ... 382/181 (73) Assignee: ABBYY Development LLC (RU) 7,716,037 B2 * 5/2010 Precoda et al. ................... TO4/2 7.912,289 B2 * 3/2011 Kansal et al. ..... ... 382/181 (*) Notice: Subject to any disclaimer, the term of this 7.917.286 B2 * 3/2011 Taylor et al. ...... ... 701 468 patent is extended or adjusted under 35 7.974,832 B2* 7/2011 Furuuchi et al. .. TO4f7 U.S.C. 154(b) by 267 days. 8,214,768 B2 * 7/2012 Boule et al. .................. T15,863 8,218,020 B2 * 7/2012 Tenchio ............ GO6F 17/30017 (21) Appl. No.: 13/651,866 348,211.3 8,239,183 B2 * 8/2012 Sembower ........................ TO4/2 (22) Filed: Oct. 15, 2012 8,312,032 B2 * 1 1/2012 Choi et al. .................... 707/758 8,315,850 B2 * 1 1/2012 Furuuchi et al. .. TO4f7 (65) Prior Publication Data 8.364464 B2 * 1/2013 Elgazzar et al. ... ... 704/2 US 2014/OO81619 A1 Mara. 20, 2014 8,373,724 B2 * 2/2013 Ratnakar et al. ............. 345,619 s (Continued) Related U.S. Application Data (63) Continuation-in-part of application No. 13/694,900, Primary Examiner — Matthew Baker filed on Sep. 18, 2012. (74) Attorney, Agent, or Firm — Veronia Weinstein (51) Int. Cl. G06F 7/28 (2006.01) (57) ABSTRACT G06F 17/30 (2006.01) (52) U.S. Cl. Methods are described for efficient and substantially instant CPC ....... G06F 17/289 (2013.01); G06F 17/30247 recognition and translation of text in photographs. A user is (2013.01); G06K 2209/01 (2013.01) able to select an area of interest for Subsequent processing. (58) Field of Classification Search Optical character recognition (OCR) may be performed on CPC. G01L 19/0212: G01L 19/005; G01L 19/0017 the wider area than that selected for determining the subject USPC .......................................................... 704/2 7 domain of the text. Translation to one or more target See application file for complete search history. languages is performed. Manual corrections may be made at various stages of processing. Variations of translation are (56) References Cited presented and made available for substitution of a word or expression in the target language. Translated text is made U.S. PATENT DOCUMENTS available for further uses or for immediate access. 5,175,684 A * 12/1992 Chong .............................. TO4/3 5,608,622 A * 3/1997 Church ... 6,047,251 A * 4/2000 Pon et al. TO4f1 36 Claims, 6 Drawing Sheets 201 Starting or launching the application Y 22 Acquiring an image from camera, memory of storage - 203 Receiving and displaying the image Choosing an area of interest on the - 204 image 205 OCR - - 206 Correction the rest of OCR 20? --> ranslation - Correction the result of translation 1 208 209 X Output US 9,519,641 B2 Page 2 (56) References Cited 2006/0253491 A1* 11/2006 Gokturk et al. ........... TO7 104.1 2008/0233980 A1 9/2008 Englund et al. U.S. PATENT DOCUMENTS 2009/022O175 A1 9, 2009 TZadok et al. ................ 382,317 2010, 0008582 A1 1/2010 Kim et al. .................... 382,177 8,380,487 B2 * 2/2013 Bastide ................. GO6F 17,289 2010.0030549 A1 2/2010 Lee ....................... G06F 1, 1626 382, 182 TO4/4 8,471,824 B2 * 6/2013 Kim et al. .................... 345,173 2010/0204979 A1* 8, 2010 Chiu et al. ........................ TO4/3 8,515,185 B2 * 8/2013 Lee ........................ GO6K 9,036 2011/0010162 A1* 1/2011 Elgazzar et al. .................. TO4/2 382,190 2011/0066421 A1 3f2011 Lee ....................... GO6F 17,289 8,626,512 B2 * 1/2014 Kurzweil et al. ............. 7041270 TO4/2 8,862,456 B2 10/2014 Krack et al. 2011/0167350 A1* 7, 2011 Hoellwarth ................... 71.5/727 8,954,314 B2 2/2015 Powalowski 2011/0313754 A1* 12/2011 Bastide et al. .................... TO4/2 8,965,126 B2 * 2/2015 Yamazoe et al. ............. 382,177 2012/0010870 A1* 1/2012 Selegey et al. ................... TO4/3 2001/0032070 A1* 10, 2001 Teicher ................... GO6F 17,28 2012/0068937 A1 3/2012 Backlund et al. ............ 345,173 TO4/2 2012/0109632 A1 5/2012 Sugiura et al. 2002/0194300 A1* 12/2002 Lin ....................... GO6F 17,289 2012fO245922 A1* 9, 2012 Kozlova et al. .................. TO4/3 709/217 2013,0004-068 A1 1/2013 KOO ...................... GO6F 17,289 2003/0200078 A1* 10, 2003 Luo et al. ......................... TO4/2 382,165 53585. A '58 TO4/2 2013/0103383 A1* 4/2013 Du et al. ........................... TO4/3 2004/O138872 A1* T/2004 Nir . 70410 2013/0108115 A1* 5, 2013 Hwang et al. 382/106 2005/002 1322 A1 1/2005 Richardson ......... GO6F 17,2836 2,392 A. 929 Pawalowski ... 704/3 TO4/2 2013/0285930 A. 10/2013 Thorsander et al. ......... 345,173 2005/0197825 A1* 9/2005 Hagerman et al. ............... TO4/2 2014/0081619 A1 3/2014 Solntseva. ............. G06F 12. 2005/0267734 Al ck 12/2005 Masuyama................. Ci 2014/0180670 A1* 6 2014 Osipova. ... 704/3 2006/0047704 A1* 3/2006 Gopalakrishnan ......... TO7 104.1 ck 2006, 0083431 A1* 4, 2006 B1 GO6F 17,289 2014/0215329 A1* 7, 2014 Zilberman et all 715/702 ISS . 2014/0222413 A1* 8, 2014 Rossmann .......... ... 704/3 382,229 2014/0297256 A1* 10/2014 Rogowski et al. ............... TO4/2 2006/0173829 A1* 8, 2006 Neeman ............................ 707/3 2006/0245005 A1* 11, 2006 Hall et al. 358,448 * cited by examiner U.S. Patent Dec. 13, 2016 Sheet 1 of 6 US 9,519,641 B2 104. O6 Muffins mit Mandeln und Schokoladen Raspeln Manden, Mehl, Backpulver, Vanitezuck vermischen. In einer anderen Schüssel Zucker und den Amaretto leichtschat danach Stück für Stück die Butter daz Danach die Eimasse Zur Mehlmasse daz und die Schokoraspen hinzufügen. Zule Masse in die Muffinformen geben und be Grad für 20-25 Min, backen. 104 Muffins with almonds and chocolate shavings Amonds, four, baking powder, vanila su. in another bow beat the eggs, sugar as frothy, then add the butter piece by pi the egg mixture to flour and add the gr mass. Finally the mixture into the muffi about 180 degrees for 20-25 minutes ba with almonds and chocolate shavings are rea Fig. 1B U.S. Patent Dec. 13, 2016 Sheet 2 of 6 US 9,519,641 B2 201 Starting or launching the application ACduiringa an image from Camera, 202 2O3 Receiving and displaying the image Choosing an area of interest on the 204 image 205 OCR 2O6 Correction the result of OCR 2O7 Translation 208 Correction the result of translation 209 Output Fig. 2 U.S. Patent Dec. 13, 2016 Sheet 3 of 6 US 9,519,641 B2 208 1. 301. Touching a screen with translated text 302 ldentifying a word or word Combination to change 303 Displaying some variants of translation or wordforms 304 Touching the best variant 305 insertion of translated word or word combination in a text Fig. 3 U.S. Patent Dec. 13, 2016 Sheet 4 of 6 US 9,519,641 B2 Muffins with almonds and chocolate shavings 1) shavings Almonds, four, baking powder, vanias In another bow beat the eggs, sugar frothy, then add the butter piece by pi the egg mixture to flour and add the g mass. Finally the mixture into the muffin about 180 degrees for 20-25 minutes ba with almonds and chocolate shavings arer AO6 Muffins with almonds and chocolate rasps Almonds, flour, baking powder, vanias In another bow beat the eggs, Sugar frothy, then add the butter piece by the egg mixture to flour and add the gra mass. Finally the mixture into the muffin about 180 degrees for 20-25 minutes ba with amonds and chocolaterasps are read Fig. 4B U.S. Patent Dec. 13, 2016 Sheet S of 6 US 9,519,641 B2 1. 205 501 Establishing the coordinates of location of a device by a navigation module 5O2 Searching in databases Country/region according to location data acquired from the navigation module 503 Searching in databases for languages that are used in a specified Country/region 504 Forming a list of languages for Subsequent recognition and related OCR functionality Fig. 5 U.S. Patent Dec. 13, 2016 Sheet 6 of 6 US 9,519,641 B2 SSSSSSS& SSSSS $33,333* -- Ss. SYYYYsssssssssssssssssssssssssssssy S Ssssssssssssssssssssssssways : MissyXYYYYYYYSSS &ss-s-s-s-s-s-s-s-s-s Ssssssssssssssssssssssssssssssssss & s Fig. 8 US 9,519,641 B2 1. 2 PHOTOGRAPHY RECOGNITION methods that generally include receiving, by an electronic TRANSLATION device, a photographic image with text to be translated, recognizing text in this photographic image with an OCR CROSS-REFERENCE TO RELATED system or technology, translating the recognized text and APPLICATIONS displaying the result of translation on the screen of elec tronic device. A user is provided with a means and an For purposes of the USPTO extra-statutory requirements, opportunity to manually correct results at each step. the present application constitutes a continuation-in-part of Using a mobile device's camera is preferable for captur U.S.