USOO8629885B2 (12) United States Patent (10) Patent No.: US 8,629,885 B2 Tzruya et al. (45) Date of Patent: Jan. 14, 2014 (54) SYSTEM, METHOD AND COMPUTER (56) References Cited PROGRAMI PRODUCT FOR DYNAMICALLY IDENTIFYING, SELECTING AND U.S. PATENT DOCUMENTS EXTRACTING GRAPHICAL AND MEDIA 5,687,376 A 11/1997 Celi et al. OBJECTS IN FRAMES OR SCENES 5,737,553 A * 4, 1998 Bartok .......................... 715,764 RENDERED BY A SOFTWARE APPLICATION (Continued) (75) Inventors: Yoav Tzruya, Even Yehuda (IL); Itay Nave, Kfar Hess (IL) FOREIGN PATENT DOCUMENTS (73) Assignee: Exent Technologies, Ltd., Petach-Tikva JP 2001-312286 A 11, 2001 (IL) KR 2005O111581 11, 2005 (*) Notice: Subject to any disclaimer, the term of this (Continued) patent is extended or adjusted under 35 U.S.C. 154(b) by 1421 days. OTHER PUBLICATIONS (21) Appl. No.: 11/588,318 Dr. Daniel E. Lipkie, Steven R. Evans, John K. Newlin, and Robert L. Weissman; Star Graphics: An Object-Oriented Implementation; Jul. (22) Filed: Oct. 27, 2006 1982; ACM; Computer Graphics; vol. 15, No. 3, pp. 115-124.* (65) Prior Publication Data (Continued) US 2007/O126749 A1 Jun. 7, 2007 Primary Examiner — Jeffery A Brier Related U.S. Application Data (74) Attorney, Agent, or Firm — Fiala & Weaver P.L.L.C. (63) Continuation-in-part of application No. 1 1/492,069, (57) ABSTRACT filed on Jul. 25, 2006, now abandoned, which is a continuation-in-part of application No. 1 1/472,454, A technique is described herein for dynamically enhancing filed on Jun. 22, 2006, now Pat. No. 7,596,536, which and measuring a software application without having to is a continuation-in-part of application No. change and recompile the original application code. A system includes a staging environment that monitors the application 11/290,830, filed on Dec. 1, 2005, now Pat. No. and indexes items of graphical and/or audio information into 7,596,540. a first database. A second database is then populated with (60) Provisional application No. 60/810,177, filed on Jun. business rules that are associated with the objects indexed. 2, 2006, provisional application No. 60/798,710, filed The system further includes a run-time environment that on May 9, 2006, provisional application No. identifies items of graphics and/or audio information gener 60/797,669, filed on May 5, 2006. ated during run-time, and upon determining that an identified (51) Int. C. item is associated with a business rule, applies the business G09G 5/00 (2006.01) rule, measures the object and its related attributes and option ally report the data back to a third database. Also described (52) U.S. C. herein are techniques for dynamically measuring attributes of USPC .......................................................... 345/619 objects rendered and/or referenced by software applications, (58) Field of Classification Search for dynamically serving advertisements to a computer game, CPC ................ A63F2300/5506; A63F2300/6009; and for pre-processing Software applications to identify spots GO6F 22O1f865 desirable for advertisement placement. USPC ........................................... 345/619; 717/113 See application file for complete search history. 22 Claims, 31 Drawing Sheets 2904 2902 \. -6 ldentify objects in the current fare as the current frame is K it 3d indsid -- p. User browses through objects in current frame Yes 290 Xa Automatically gather information or the selected object 292 Allow use to provide additional information on the selected object captional) 294 2 Stors information US 8,629,885 B2 Page 2 (56) References Cited 2007.0143603 A1 6/2007 Hadden et al. 2007. O168309 A1 7/2007 Tzruya et al. U.S. PATENT DOCUMENTS 2007/0296718 A1 12/2007 Tzruya et al. 2008.0009344 A1 1/2008 Graham et al. 5,905,492 A 5, 1999 Straub et al. 2009/0307173 A1 12/2009 Tzruya et al. 5.991,836 A 11/1999 Renda 2010.0036785 A1 2/2010 Tzruya et al. 6,021.438 A 2/2000 Duvvoori et al. 2011/0173054 A1 7/2011 Kutaragi et al. 6,036,601 A 3, 2000 Heckel 2012/0054781 A1 3/2012 Tzruya et al. 6,047,123 A 4/2000 Brown et al. 2012/0291.032 A1 1 1/2012 Tzruya et al. 6,134,597 A 10, 2000 Rieth et al. 6,163,317 A 12/2000 de Judicibus FOREIGN PATENT DOCUMENTS 6,202,058 B1 3, 2001 Rose et al. 6,278,966 B1 8, 2001 Howard et al. WO 9115921 A1 10, 1991 6,311,221 B1 10/2001 Raz et al. WO 952.5399 A1 9, 1995 6,3 14,470 B1 11/2001 Ward et al. WO 99.24904 A1 5, 1999 6,330,711 B1 12/2001 Knutson WO OO29964 A1 5, 2000 6,539,544 B2 3/2003 Ebisawa WO 2004/0750O2 A2 9, 2004 6,616,533 B1 9, 2003 Rashkovskiy WO 2005096782 A2 10/2005 6,631423 B1 10/2003 Brown et al. WO 2007063422 A2 6, 2007 6,785,659 B1 8/2004 Landsman et al. WO 2007063422 A3 6, 2007 6,802,055 B2 10, 2004 Jade et al. WO 2007 148233 A2 12/2007 6,868,525 B1 3/2005 Szabo WO 2007 148233 A3 12/2007 6,907,566 B1 6, 2005 McElfresh et al. WO 2008020313 A2 2, 2008 6,954,728 B1 10/2005 Kusumoto et al. WO 2008020313 A3 2, 2008 7,003,781 B1 2/2006 Blackwell et al. WO 2008020317 A2 2, 2008 7,076.445 B1 T/2006 Cartwright WO 2008020317 A3 2, 2008 7,076.736 B2 7/2006 Hugh WO 2008104834 A2 9, 2008 7,120,619 B2 10, 2006 Drucker et al. WO 2008104834 A3 9, 2008 7,246,254 B2 7/2007 Alur et al. WO 2009034478 A2 3, 2009 7,249,140 B1 7/2007 Korenevsky et al. WO 2009034478 A3 3, 2009 7,436,406 B2 * 10/2008 Dautelle ....................... 345,441 7,451,187 B2 11/2008 Malik et al. 7.487,112 B2 2/2009 Barnes, Jr. OTHER PUBLICATIONS 7,596,536 B2 9/2009 TZruya et al. 7,596,540 B2 9/2009 TZruya et al. Mohr, et al., “HijackGL: Reconstructing from Streams for Stylized 7,818,691 B2 10/2010 Irvine Rendering”. Association for Computing Machinery, Inc., (Jun. 3, 8,060,460 B2 11/2011 Tzruya et al. 2002 13-20 8,069,136 B2 11/2011 Tzruya et al. ), pp. 13-20. 8,214,256 B2 7, 2012 Riedlet al. Walders, H., "glAnalyze The Newest OpenGL Product'. http:// 2002fOOO2568 A1 1/2002 Judson www.3dsite.com/n? sites/3dsite/newsletterfissues/10/sec12.html, 2002fOO38344 A1 3, 2002 Ullman et al. retrieved Apr. 15, 2002, (Apr. 29, 1999), pp. 1-3. 2002/0040322 A1 4/2002 Ebisawa “New In-Game Advertising Solution from Exent Technologies Pro 2002.0099837 A1 7, 2002 Oeet al. New ng olog 2002fO112033 A1 8/2002 Doemling et al. vides Dramatic Increase in Ad Inventory Without Technical Integra 2002/0129349 A1 9, 2002 Ebisawa tion'. http://www.exent.com/news/pr/20060427.asp, retrieved Mar. 2002/0147858 A1 10/2002 Motoyama et al. 14, 2008, (Apr. 27, 2006), pp. 1-2. 2002fO154214 A1 10, 2002 Scallie et al. “Exent Technologies: In-Game Advertising”.http://web.archive.org/ 2002/0178302 A1 1 1/2002 Tracey web/20060504072829/http://exent.com/products and services/ 388 A. 53. all In Game Advertising.asp, retrieved Mar. 14, 2008, (May 4, 2006), 2003/013 1286 A1 7/2003 Kaler et al. 1 page. 2003.0167202 A1 9, 2003 Marks et al. Schu, M. et al., “System-on-Silicon solution for High Quality Con 2003/0204275 A1 10, 2003 Krubeck Sumer Video Processing The Next Generation', IEEE Transactions 2003/0208754 A1 11, 2003 Sridhar et al. on Consumer Electronics, IEEE Service Center, New York, NY, US, 38 A .338 Risler 70 vol. 47, No. 3, (Aug. 1, 2001), pp. 412-419. 2004/011.6183 A1 6, 2004 SE" Ca Chalil, M. et al., “MPEG-4 Based Framework for Game Engine to 2004/O12294.0 A1 6, 2004 Gibson et al. Handle Virtual Advertisements in Games'. International Conference 2004/O133876 A1 7/2004 Sproule on Multimedia and Expo, (Jul. 6, 2003), 413-416. 2004/O148221 A1 7, 2004 Chu Office Action Received for European Patent Application No. 38: A. 33.9 t et al. 08830539.6, mailed on Jun. 10, 2010, 3 pages. 2004/0217987 A1 11, 2004 title Office Action Received for European Patent Application No. 2005, OO15641 A1 1/2005 Alur et al. 07872836.7, mailed on Jul 19, 2007, 3 pages. 2005.0068567 A1 3/2005 Hull et al. International Search Report and Written Opinion received for PCT 2005/0223355 A1* 10/2005 Forstmann et al. ........... 717,115 Patent Application No. PCT/IB2006/003968, mailed on Aug. 30, 2005.0246174 A1 11, 2005 DeGolia 2007, 14 pages. 2006, OO85812 A1 4/2006 Shishegar et al. 2006.0128469 A1 6, 2006 Willis et al. International Preliminary Report on Patentability Received for PCT 2006.0143675 A1 6, 2006 Willis et al. Patent Application No. PCT/IB2006/003968, mailed on Jun. 12, 2006, O155643 A1 7/2006 Shuman et al. 2008, 8 pages. 2006, O190429 A1 8/2006 Sidlosky et al. Office Action Received for European Patent Application No. 22999. A 299. Singhe 07825369.7, mailed on Mar. 13, 2009, 6 pages. 2007/0061201 A1 3, 2007 A. Office Action Received for European Patent Application No. 2007/OO72676 A1 3/2007 Baluja 07825369.7, mailed on Jan. 18, 2010, 6 pages. 2007/0129146 A1 6/2007 Tzruya et al. Office Action Received for Korean Patent Application No. 10-2008 2007/012999.0 A1 6/2007 Tzruya et al.