(12) Patent Application Publication (10) Pub. No.: US 2013/0273968 A1 Rhoads Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US 20130273968A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2013/0273968 A1 Rhoads et al. (43) Pub. Date: Oct. 17, 2013 (54) METHODS AND SYSTEMS FOR CONTENT Publication Classification PROCESSING (51) Int. Cl. (71) Applicant: Digimarc Corporation, (US) H04M I/02 (2006.01) (52) U.S. Cl. (72) Inventors: Geoffrey B. Rhoads, West Linn, OR CPC ................................... H04M I/0264 (2013.01) (US); Tony F. Rodriguez, Portland, OR USPC ......... 455/556.1; 382/305; 382/103:455/566 (US) (57) ABSTRACT Cell phones and other portable devices are equipped with a (21) Appl. No.: 13/774,056 variety oftechnologies by which existing functionality can be improved, and new functionality can be provided. Some (22) Filed: Feb. 22, 2013 aspects relate to data driven imaging architectures, in which a 9 cell phone's image sensor is one in a chain of stages that Successively act on packetized instructions/data, to capture Related U.S. Application Data and later process imagery. Other aspects relate to distribution (63) Continuation of application No. 12/484.115, filed on of processing tasks between the device and remote resources Jun. 12, 2009, now Pat. No. s 385 97 which is a (“the cloud'). Elemental Image processing, Such as filtering COtinuation-in-p art of application No. 12A271,692 and edge detection and even some simpler template match filed on Nov. 14, 2008, now Pat. No. 8,520,979 sw- as ing operations—may be performed on the cell phone. Other • u is s • L vs. 8 Y-sa-1 sys- - • operations are referred out to remote service providers. The (60) Provisional application No. 61/090,083, filed on Aug. remote service providers can be identified using techniques 19, 2008, provisional application No. 61/096,703, Such as a reverse auction, though which they compete for filed on Sep. 12, 2008, provisional application No. processing tasks. Other aspects of the disclosed technologies 61/100,643, filed on Sep. 26, 2008, provisional appli relate to visual search capabilities, and determining appropri cation No. 61/103,907, filed on Oct. 8, 2008, provi ate actions responsive to different image inputs. Still others sional application No. 61/110,490, filed on Oct. 31, concern metadata generation, processing, and representation. 2008, provisional application No. 61/176.739, filed on Yet others relate to coping with fixed focus limitations of cell May 8, 2009, provisional application No. 61/174,822, phone cameras, e.g., in reading digital watermark data. Still filed on May 1, 2009, provisional application No. others concern user interface improvements. A great number 61/169.266, filed on Apr. 14, 2009. of other features and arrangements are also detailed. CELL NETWORK TO COMM SERVICES CHANNEL INTERFACE WIFI TO INTERNET TRAFFIC MONITORING 8. BILLING SYSTEM Patent Application Publication Oct. 17, 2013 Sheet 1 of 52 US 2013/0273968A1 O|X|\,|ONALENTITEO WELSÅSSONITTIE Patent Application Publication Oct. 17, 2013 Sheet 2 of 52 US 2013/0273968A1 EENAN\/WITH HOSEEHEDEC]) (NOILVHV?ES LOETEO ERHT,LSE|5) Patent Application Publication Oct. 17, 2013 Sheet 3 of 52 US 2013/0273968A1 LINEW/\\/c) SY-HOSSEOORHCH SSEITERHIWA ETSOOOSO NESTEIN Patent Application Publication Oct. 17, 2013 Sheet 4 of 52 US 2013/0273968A1 KEYVECTOR DATA (PIXELS & DERIVATIVES) FIG. 4 / N RESULTA RESULT B RESULT C KEYVECTOR FIG. 4A ADDRESSING, ETC / PXEL PACKET Patent Application Publication Oct. 17, 2013 Sheet 5 of 52 US 2013/0273968A1 FIG. 4B HEADER DATA KEYVECTOR DATA TASKA TASK B TASK C TASKD VISUALTASK COMMONALITY PE CHARTS H FFT % RE-SAMPLING 1 PDF417 BARCODE READING FIG. 5 Patent Application Publication Oct. 17, 2013 Sheet 6 of 52 US 2013/0273968A1 RESIDENT CALL-UP VISUAL PROCESSING SERVICES ON ON OFF OFF ON OFF COMMON SERVICES SORTER LEVEL PXEL PRO CESSING LIBRARY FLOW GATE CONFIGURATION, SOFTWARE PROGRAMMING EXTERNAL PXEL KEYVECTOR SENSOR Msier HARDWARE processess-ROUTINGDATA OTHER) FIG. 6 INTERNAL Patent Application Publication Oct. 17, 2013 Sheet 7 of 52 US 2013/0273968A1 LNALN OTENNWHOWWOO STWOO Patent Application Publication Oct. 17, 2013 Sheet 8 of 52 US 2013/0273968A1 BNAS LN Sng WoO Patent Application Publication Oct. 17, 2013 Sheet 9 of 52 US 2013/0273968A1 SOAO Patent Application Publication Oct. 17, 2013 Sheet 10 of 52 US 2013/0273968A1 \/LVCI·HEYA‘HOSSEOOHd ENOHc]TTEO Patent Application Publication Oct. 17, 2013 Sheet 11 of 52 US 2013/0273968A1 EOLAECI/TV/OOT Patent Application Publication Oct. 17, 2013 Sheet 12 of 52 US 2013/0273968A1 MY PROFILE, INCLUDING WHO IAM, AND WHAT DO GENERALLY LIKEIWANT MY CONTEXT, INCLUDING WHERE IAM, AND WHAT ARE MY CURRENT DESRES MY VISUAL OUERY PIXELS THEMSELVES PACKETS RESULTS VISUAL CRUERY PACKET PACKETS RESULTS PACKETS RESULTS CONFIGURABLE ARRAY OF OFFEREDATAUCTION TO AN SERVICE PROVIDERS ARRAY OF SERVICE PROVIDERS FIG. 11 Patent Application Publication Oct. 17, 2013 Sheet 13 of 52 US 2013/0273968A1 HØIH‘XOITTO ESNOCHSENHETTV/A S_LITTISENH EOLAECIRJEST Patent Application Publication Oct. 17, 2013 Sheet 14 of 52 US 2013/0273968A1 12 SENSOR PROCESSING FOR HUMAN VISUAL | SYSTEM (e.g., JPEG compression) FIG. 13 16 56 CAMERA STAGE 1 STAGE 2 STAGE Y INSTRUCTIONS INSTRUCTIONS INSTRUCTIONS ' ' ' INSTRUCTIONS 55 58a 58b. STAGEZ INSTRUCTIONS DATA 56 59 FIG. 17 Patent Application Publication Oct. 17, 2013 Sheet 15 of 52 US 2013/0273968A1 CELL PHONE FIG. 14 ADDRESSING 15 APPLICATION IN CELL PROCESSING PHONE EXTERNAL FROM CELL PHONE APPLICATION IN CELL l PHONE APPLICATION - - - - - - - - - - - - - - - - - - - - - EXTERNAL FROM 1O CELL PHONE APPLICATION EXTERNAL FROM CELL PHONE Patent Application Publication Oct. 17, 2013 Sheet 16 of 52 US 2013/0273968A1 S. Patent Application Publication Oct. 17, 2013 Sheet 17 of 52 US 2013/0273968A1 0 9 .|y=evis(??|c??vis 999 |9 Patent Application Publication Oct. 17, 2013 Sheet 18 of 52 US 2013/0273968A1 ERHVWAL-IOS 8),"SOIH (ZZ ERHV/AACIN-IV/H Patent Application Publication Oct. 17, 2013 Sheet 20 of 52 US 2013/0273968A1 MICRO MIRROR PROJECTOR Patent Application Publication Oct. 17, 2013 Sheet 21 of 52 US 2013/0273968A1 Patent Application Publication Oct. 17, 2013 Sheet 22 of 52 US 2013/0273968A1 CAPTURE OR RECEIVE IMAGE IMAGE AND/OR METADATA PRE-PROCESS IMAGE, SUBMIT TO GOOGLE, RECEIVE e.g., IMAGE ENHANCEMENT, AUXLIARY INFORMATION OBJECT SEGMENTATION/ L- EXTRACTION, etc. SUBMIT TO SERVICE FIG. 24 CAPTURE OR RECEIVE IMAGE FEATURE EXTRACTION SEARCH FOR IMAGES WITH AUGMENTENHANCE | SIMILAR METRICS METADATA USE SIMLARMAGE DATAN SUBMIT TO SERVICE LIEU OF USER IMAGE DATA FIG. 25 FIG. 23 Patent Application Publication Oct. 17, 2013 Sheet 23 of 52 US 2013/0273968A1 CAPTURE OR RECEIVE IMAGE CAPTURE OR RECEIVE IMAGE FEATURE EXTRACTION FEATURE EXTRACTION SEARCH FOR IMAGES WITH SEARCH FOR IMAGES WITH SIMILAR METRICS SIMILAR METRICS SUBMIT PLURAL SETS OF COMPOSITE OR MERGE IMAGE DATA TO SERVICE SEVERAL SETS OF IMAGE DATA PROVIDER SUBMIT TO SERVICE PROVIDER SERVICE PROVIDER, OR USER DEVICE, DETERMINES ENHANCED RESPONSE BASED ON PLURAL SETS OF DATA FIG. 27 FIG. 26 CAPTURE OR RECEIVE USER IMAGE FEATURE EXTRACTION SEARCH FOR IMAGES WITH SIMILAR METRICS USE INFO FROM SIMILAR IMAGE(S) TO ENHANCE USER IMAGE FIG. 28 Patent Application Publication Oct. 17, 2013 Sheet 24 of 52 US 2013/0273968A1 CAPTURE OR RECEIVE IMAGE CAPTURE OR RECEIVE IMAGE OF SUBJECT WITH GEOLOCATION INFO FEATURE EXTRACTION INPUT GEOLOCATION INFO TO SEARCH FOR IMAGES WITH DB; GET TEXTUAL SIMILAR METRICS DESCRIPTORS HARVEST METADATA SEARCH IMAGE DATABASE USING TEXTUAL DESCRIPTORS, IDENTIFY OTHER IMAGE(S) OF SUBJECT SEARCH FOR IMAGES WITH SIMILARMETADATA SUBMIT TO SERVICE SUBMIT TO SERVICE FIG. 30 FIG. 28A CAPTURE OR RECEIVE IMAGE OF SUBJECT WITH GEOLOCATION INFO SEARCH IMAGE DATABASE USING GEOLOCATION DATA IDENTIFY OTHER IMAGE(S) OF SUBJECT SUBMIT OTHER IMAGE(S) TO SERVICE FIG. 31 Patent Application Publication Oct. 17, 2013 Sheet 25 of 52 US 2013/0273968A1 FIG. 36 FIG. 37 FIG. 38 Patent Application Publication Oct. 17, 2013 Sheet 26 of 52 US 2013/0273968A1 CAPTURE OR RECEIVE IMAGE CAPTURE OR RECEIVE IMAGE OF SUBJECT WITH OF SUBJECT WITH GEOLOCATION INFO GEOLOCATION INFO SEARCH FOR IMAGES WITH SEARCH FOR IMAGES WITH SIMILAR IMAGE METRICS SIMILAR IMAGE METRICS ADD GEOLOCATION INFO TO HARVEST METADATA MATCHING IMAGES FIG. 32 SEARCH FOR IMAGES WITH SIMILAR METADATA ADD GEOLOCATION INFO TO CAPTURE OR RECEIVE IMAGE MATCHING IMAGES OF SUBJECT WITH GEOLOCATION INFO FIG. 33 SEARCH FOR IMAGES WITH MATCHING GEOLOCATION HARVEST METADATA SEARCH FOR IMAGES WITH SIMILAR METADATA ADD GEOLOCATION INFO TO MATCHING IMAGES FIG. 34 Patent Application Publication Oct. 17, 2013 Sheet 27 of 52 US 2013/0273968A1 USER SNAPS IMAGE URBANIRURAL FAMILYIFRIEND/ PRSSSFFER ANALYSIS STRANGER ANALYSIS ANALYSIS RESPONSE RESPONSE RESPONSE ENGINEE ENGINE #2 ENGINE #3 FIG. 49 CONSOLIDATOR CELL PHONE USER INTERFACE ROUTER RESPONSE RESPONSE ENGINEA ENGINEE FIG. 37A FIG. 37B Patent Application Publication Oct. 17, 2013 Sheet 28 of 52 US 2013/0273968A1 USER-IMAGE -o "IMAGE JUICER" (algorithmic, Crowd-sourced, etc.) USER IMAGE-RELATED FACTORS (First Tier) e.g., data about one or more of: Color histogram, Pattern, Shape, Texture, Facial recognition, Eigenvalue, Object recognition, Orientation, Text, Numbers, OCR, Barcode, Watermark, Emotion, Location, Transform data, etc DATABASE OF PUBLIC IMAGES, SEARCHENGINE --> IMAGE-RELATED FACTORS, AND OTHER METADATA OTHER IMAGE-RELATED OTHER FACTORS AND/OR IMAGESICONTENT OTHER METADATA (Second Tier) (Second Tier) - - - - - - DATABASE OF PUBLIC IMAGES, SEARCHENGINE ---> IMAGE-RELATED FACTORS, AND - L- OTHER METADATA FINAL FINAL OTHER IMAGESICONTENT INFORMATION (Nth Tier) (Nth Tier) USER FIG. 39 Patent Application Publication Oct. 17, 2013 Sheet 29 of 52 US 2013/0273968A1 USER-IMAGE -o "IMAGE JUICER" (algorithmic, Crowd-sourced, etc.) USER IMAGE-RELATED FACTORS (First Tier) DATABASE OF PUBLIC IMAGES, SEARCHENGINE - e IMAGE-RELATED FACTORS, AND OTHER METADATA OTHER IMAGE-RELATED OTHER FACTORS AND/OR IMAGESICONTENT OTHER METADATA (Second Tier) (Second Tier) GOOGLETEXT, SEARCHENGINE Ho- WEB DATABASE - - - - - - - - DATABASE OF PUBLIC IMAGES, SEARCHENGINE ---> IMAGE-RELATED FACTORS, AND - - - - - - - - - - - -OTHER - METADATA- - - - - FINAL FINAL OTHER IMAGESICONTENT INFORMATION (Nth Tier) (Nth Tier) FIG.