(12) United States Patent (10) Patent No.: US 8,805,110 B2 Rhoads Et Al
Total Page:16
File Type:pdf, Size:1020Kb
USOO88051 1 OB2 (12) United States Patent (10) Patent No.: US 8,805,110 B2 Rhoads et al. (45) Date of Patent: Aug. 12, 2014 (54) METHODS AND SYSTEMS FOR CONTENT (56) References Cited PROCESSING (75) Inventors: Geoffrey B. Rhoads, West Linn, OR U.S. PATENT DOCUMENTS (US); Tony F. Rodriguez, Portland, OR (US); John D. Lord, West Linn, OR 5,835,616 A * 1 1/1998 Lobo et al. .................... 382,118 (US); Brian T. MacIntosh, Lake 6,301,370 B1 * 10/2001 Steffens et al. ............... 382,103 Oswego, OR (US); Nicole Rhoads, West (Continued) Linn, OR (US); William Y. Conwell, OTHER PUBLICATIONS Portland, OR (US) (73) Assignee: Digimarc Corporation, Beaverton, OR GPU Computing: Graphics Processing Units- powerful, program (US) mable, and highly parallel—are increasingly targeting general-pur (*) Notice: Subject to any disclaimer, the term of this pose computing applications, by John D. Owens, Mike Houston, patent is extended or adjusted under 35 David Luebke, Simon Green, John E. Stone, and James C. Phillips, U.S.C. 154(b) by 515 days. 0018-9219, 2008 IEEE vol. 96, No. 5, pp. 279-299, May 2008 | Proceedings of the IEEE.* (21) Appl. No.: 13/011,618 (22) Filed: Jan. 21, 2011 (Continued) (65) Prior Publication Data Primary Examiner — Hadi Akhavannik US 2011 FO212717 A1 Sep. 1, 2011 Assistant Examiner — Mehdi Rashidian (74) Attorney, Agent, or Firm — Digimarc Corporation Related U.S. Application Data (63) Continuation of application No. (57) ABSTRACT PCT/US2009/054358, filed on Aug. 19, 2009, and a Mobile phones and other portable devices are equipped with a variety of technologies by which existing functionality can (Continued) be improved, and new functionality can be provided. Some (51) Int. C. aspects relate to visual search capabilities, and determining G06K 9/00 (2006.01) appropriate actions responsive to different image inputs. Oth HO4M 3/OO (2006.01) ers relate to processing of image data. Still others concern HO4N 13/OO (2006.01) metadata generation, processing, and representation. Yet oth GO6T 1/20 (2006.01) (52) U.S. C. ers concern user interface improvements. Other aspects relate CPC .......... G06K 9/00 (2013.01); H04N 2013/0074 to imaging architectures, in which a mobile phone's image (2013.01); G06T 1/20 (2013.01) sensor is one in a chain of stages that successively act on USPC ............................ 382/255; 455/418; 455/420 packetized instructions/data, to capture and later process (58) Field of Classification Search imagery. Still other aspects relate to distribution of processing CPC ...... G06K9/00; G06K9/0021; G06T 1/0007; tasks between the mobile device and remote resources (“the G06T 1/20: H04N 2013/0074 cloud'). Elemental image processing (e.g., simple filtering USPC ......... 382/100, 103,106, 107, 115, 117, 118, and edge detection) can be performed on the mobile phone, 382/190, 199, 209, 255, 276:348/61, while other operations can be referred out to remote service 348/143-160,273, 376,208.15, providers. The remote service providers can be selected using 348/333.01-333.04, 333.11, 240.99, techniques such as reverse auctions, through which they com 348/240.03, 208.4; 455/403, 418, 420; pete for processing tasks. A great number of other features 356/3; 726/17: 707/687, 690, 769, and arrangements are also detailed. 707/770 771 See application file for complete search history. 9 Claims, 60 Drawing Sheets RERIEWE MANAL GESRE FACIAL &"gif RECOGNITION OBJECT RECOGNITC NEARBY? POST TO BARCODE AC3ook IST OR RADING SAE ON CRAGSIST TRANSLAE iS^ OENGIS SCOWER HISTORY HUMAN WEB COMMAND (DEGREES OF ACCESS (-) Ntry SEPARATION) /AUTENTATION US 8,805,110 B2 Page 2 Related U.S. Application Data (56) References Cited continuation-in-part of application No. 12/271,692, U.S. PATENT DOCUMENTS filed on Nov. 14, 2008, now Pat. No. 8,520,979, and a continuation-in-part of application No. 12/484,115, 7,742,624 B2* 6/2010 Super et al. ................... 382/106 filed on Jun. 12, 2009, now Pat. No. 8,385,971, and a 8.467,627 B2 6, 2013 Gwak et al. .... ... 382.275 continuation-in-part of application No. 12/498,709, 2002fOO19819 A1* 2/2002 Sekiguchi et al. ................ 707/3 filed on Jul. 7, 2009. 2004/0263663 A1: 12/2004 Lee et al. ........... ... 348,333.11 2006, OO 12677 A1 1/2006 Neven et al. .................... 348.61 (60) Provisional application No. 61/090,083, filed on Aug. 2007/0162971 A1* 7/2007 Blom et al. ..................... 726/17 19, 2008, provisional application No. 61/096,703, 2007/0248281 A1* 10/2007 Super et al. ... ... 382.275 filed on Sep. 12, 2008, provisional application No. 2008.OOOT 620 A1* 1/2008 Wang et al. ... ... 348,154 61/100,643, filed on Sep. 26, 2008, provisional 2009/0080698 A1 3f2009 Mihara et al. .. ... 38.2/103 application No. 61/103,907, filed on Oct. 8, 2008, 2011/O116720 Al 5/2011 Gwak et al. ................... 382,224 provisional application No. 61/110,490, filed on Oct. 31, 2008, provisional application No. 61/169,266, OTHER PUBLICATIONS filed on Apr. 14, 2009, provisional application No. 61/174,822, filed on May 1, 2009, provisional Yang et al. Detecting Faces in Images: A Survey, IEEE Transactions application No. 61/176.739, filed on May 8, 2009, on Pattern Analysis and Machine Intelligence, vol. 24. No. 1, Jan. provisional application No. 61/226,195, filed on Jul. 2002. 16, 2009, provisional application No. 61/234,542, filed on Aug. 17, 2009. * cited by examiner U.S. Patent Aug. 12, 2014 Sheet 1 of 60 US 8,805,110 B2 U.S. Patent Aug. 12, 2014 Sheet 2 of 60 US 8,805,110 B2 OLXAHONALENTITEO LENHELNIOLHIM|—?z?– U.S. Patent Aug. 12, 2014 Sheet 3 of 60 US 8,805,110 B2 SSEOOV/(NOILVAJV?ES LOETEO ERHfI_LSESO U.S. Patent Aug. 12, 2014 Sheet 4 of 60 US 8,805,110 B2 ERHTMLOTT,HLS SèHOSSEOO?Ho-H AO EHL9."SOIH8HEMA U.S. Patent Aug. 12, 2014 Sheet 5 of 60 US 8,805,110 B2 KEYVECTOR DATA (PIXELS & DERIVATIVES) FIG. 4 VISUAL TASK B RESULTA RESULT B RESULT C KEYVECTOR FIG. 4A ADDRESSING, ETC PXEL PACKET U.S. Patent Aug. 12, 2014 Sheet 6 of 60 US 8,805,110 B2 FIG. 4B VISUALTASK COMMONALITY PE CHARTS I FFT % RE-SAMPLNG a PDF417 BARCODE READING FIG. 5 U.S. Patent Aug. 12, 2014 Sheet 7 of 60 US 8,805,110 B2 RESIDENT CALL-UP VISUAL PROCESSING SERVICES ON ON OFF OFF ON OFF COMMON SERVICES SORTER EYEE 3H:2 xNS Š CESSINGPRO 2H:N.xxx xxxssess 2.:::S FLOW GATE CONFIGURATION, SOFTWARE PROGRAMMING EXTERNAL PXEL KEYVECTOR SENSOR milier HARDWARE pRoc,DATA Exelss ROUTING OTHER) FIG. 6 INTERNAL U.S. Patent US 8,805,110 B2 U.S. Patent Aug. 12, 2014 Sheet 10 of 60 US 8,805,110 B2 SOAO U.S. Patent US 8,805,110 B2 U.S. Patent Aug. 12, 2014 Sheet 12 of 60 US 8,805,110 B2 U.S. Patent Aug. 12, 2014 Sheet 13 of 60 US 8,805,110 B2 MY PROFILE, INCLUDING WHO AM, AND WHAT DO GENERALLY LIKEWANT MY CONTEXT, INCLUDING WHERE I AM, AND WHAT ARE MY CURRENT DESRES MY VISUAL QUERY PXELS THEMSELVES PACKETS RESULTS VISUAL OUERY PACKET PACKETS RESULTS PACKETS RESULTS CONFIGURABLE ARRAY OF OFFEREDATAUCTION TO AN SERVICE PROVIDERS ARRAY OF SERVICE PROVIDERS FIG. 11 U.S. Patent S_LITTISERH EOLAECIMJEST U.S. Patent Aug. 12, 2014 Sheet 15 of 60 US 8,805,110 B2 12 SENSOR PROCESSING FOR HUMAN VISUAL SYSTEM (e.g., JPEG compression) No FIG. 13 16 57 Y 56 - N CAMERA STAGE 1 STAGE 2 STAGE Y INSTRUCTIONS INSTRUCTIONS INSTRUCTIONS ' ' ' INSTRUCTIONS 55 58a 58b STAGE Z INSTRUCTIONS DATA 56 59 FIG. 17 U.S. Patent Aug. 12, 2014 Sheet 16 of 60 US 8,805,110 B2 CELL PHONE FIG. 14 APPLICATION PROCESSING NCELL EXTERNAL FROM CELL PHONE APPLICATION IN CELL APPLICATION EXTERNAL FROM CELL PHONE APPLICATION EXTERNAL FROM CELL PHONE U.S. Patent Aug. 12, 2014 Sheet 17 of 60 US 8,805,110 B2 S. U.S. Patent Aug. 12, 2014 Sheet 18 of 60 US 8,805,110 B2 v li (D CC H CO el N cy 9 H CO wa (1)||No.!!! |9 ; : : -- i | TO?]]NOO ?HOSSEOORHd U.S. Patent Aug. 12, 2014 Sheet 19 of 60 US 8,805,110 B2 9/ 8)," U.S. Patent US 8,805,110 B2 cd U.S. Patent Aug. 12, 2014 Sheet 22 of 60 US 8,805,110 B2 RESPONSE TIME ROUTING NEEDED CONSTRANTS OPERATION DETALS USER HARDWARE • High level PREFERENCES o What special purpose operations hardware is loca? O LOW level o What is Current operations hardware utilization? o Common operations o CPU pipeline length O Operations that are preconditions to a Pipeline stall risks others CONNECTIVITY o Power consumption STATUS GEOGRAPHICAL RULES, CONSIDERATIONS HEURiSTICS INFORE REMOTE PROVIDER(S), E.G., Readiness Speed Cost Attributes of importance to user OPERATIONS FOR OPERATIONS FOR LOCAL PROCESSING REMOTE PROCESSING FIG. 19B U.S. Patent Aug. 12, 2014 Sheet 23 of 60 US 8,805,110 B2 12 / CAMERA SENSOR 82 83 86 MICRO MIRROR PROJECTOR U.S. Patent Aug. 12, 2014 Sheet 24 of 60 US 8,805,110 B2 uOleOSSeO effieu Jey SSes) UOOU ul-Idueful |-– OO Jepee epOO Je9 O2/Ol Xueue) eM -1 1--1ENOHCHOMHO|W U.S. Patent Aug. 12, 2014 Sheet 25 of 60 US 8,805,110 B2 Fle 21 FIG. 22 U.S. Patent Aug. 12, 2014 Sheet 26 of 60 US 8,805,110 B2 CAPTURE OR RECEIVE IMAGE IMAGE AND/OR METADATA H.