Real-Time Chromakey Matting Using Image Statistics
Total Page:16
File Type:pdf, Size:1020Kb
University of Central Florida STARS UCF Patents Technology Transfer 7-2-2013 Real-Time Chromakey Matting Using Image Statistics Charles Hughes University of Central Florida Nicholas Beato University of Central Florida Mark Colbert University of Central Florida Kazumasa Yamazawa NAIST-Nara Institute of Science and Technology Yunjun Zhang University of Central Florida Find similar works at: https://stars.library.ucf.edu/patents University of Central Florida Libraries http://library.ucf.edu This Patent is brought to you for free and open access by the Technology Transfer at STARS. It has been accepted for inclusion in UCF Patents by an authorized administrator of STARS. For more information, please contact [email protected]. Recommended Citation Hughes, Charles; Beato, Nicholas; Colbert, Mark; Yamazawa, Kazumasa; and Zhang, Yunjun, "Real-Time Chromakey Matting Using Image Statistics" (2013). UCF Patents. 490. https://stars.library.ucf.edu/patents/490 I lllll llllllll Ill lllll lllll lllll lllll lllll 111111111111111111111111111111111 US0084 77149B2 c12) United States Patent (10) Patent No.: US 8,477,149 B2 Beato et al. (45) Date of Patent: Jul. 2, 2013 (54) REAL-TIME CHROMAKEY MATTING USING 348/586; 348/592; 358/463; 358/525; 358/540; IMAGE STATISTICS 382/159; 382/228; 382/254; 382/272; 382/300 (75) Inventors: Nicholas Beato, Orlando, FL (US); (58) Field of Classification Search Charles E. Hughes, Maitland, FL (US); USPC ................. 386/201, 224, 271, 273-274, 263, Mark Colbert, San Mateo, CA (US); 386/300, 353; 715/756-757; 345/418-419, Yunjun Zhang, Orlando, FL (US); 345/427,581,586, 589-590, 592, 600,606, Kazumasa Yamazawa, Nara (JP) 345/619,620,623-626, 629-630, 632-633, (73) Assignee: University of Central Florida Research 345/634, 638-640, 643-644,672-673;348/115, Foundation, Inc., Orlando, FL (US) 348/169, 180-182, 208.99, 208.14, 533-539, ( * ) Notice: Subject to any disclaimer, the term ofthis 348/553,557,560, 563-565, 571, 578,580, patent is extended or adjusted under 35 348/582,584-586, 587, 590-592, 597-599, U.S.C. 154(b) by 513 days. 348/607, 625, 683, 696; 358/504, 518, 525, 358/533, 537, 540,447-448,461,463;382/162, (21) Appl. No.: 121752,238 382/167,224,228,254,272,274,276,285, 382/300, 154, 159 (22) Filed: Apr. 1, 2010 See application file for complete search history. (65) Prior Publication Data (56) References Cited US 2010/0277471 Al Nov. 4, 2010 U.S. PATENT DOCUMENTS Related U.S. Application Data 5,355,174 A * 10/1994 Mishima ....................... 348/592 6,034,739 A * 3/2000 Rohlfing et al. .............. 348/586 (60) Provisional application No. 61/165,740, filed on Apr. (Continued) 1, 2009, provisional application No. 61/318,336, filed on Mar. 28, 2010. Primary Examiner - Wesner Sajous (74) Attorney, Agent, or Firm - Timothy H. Van Dyke; (51) Int. Cl. Beusse, Wolter, Sanks, Mora & Maire, P.A. G09G5/00 (2006.01) G09G5/02 (2006.01) (57) ABSTRACT G03F 3108 (2006.01) A method, system and computer readable media for real-time H04N 1138 (2006.01) chromakey matting using image statistics. To identify the H04N 1140 (2006.01) chroma key spectrum, the system/method executes in three H04N9/74 (2006.01) stages. In an off-line training stage, the system performs H04N9/75 (2006.01) semi-automatic calibration of the chroma key parameteriza H04N5/445 (2011.01) tion. In the real-time classification stage, the system estimates G06K9/40 (2006.01) the alpha matte on a GPU. Finally, an optional error minimi G06K9/62 (2006.01) zation stage improves the estimated matte, accounting for G06K9/38 (2006.01) misclassifications and signal noise. Given the resulting matte, G06K9/32 (2006.01) standard alpha blending composites the virtual scene with the (52) U.S. Cl. video feed to create the illusion that both worlds coexist. USPC ........... 345/592; 345/606; 345/629; 345/633; 345/640; 345/673; 348/115; 348/180; 348/565; 20 Claims, 6 Drawing Sheets FOREGROUND FOREGROUND v-11 VIDEO SUPPRESSION FORE 19 FOREGROUND FG KEY KEY 23 GENERATOR 21 SUM BACKGROUND BG KEY KEY GENERATOR BACKGROUND BACKGROUND VIDEO SUPPRESSION 14 US 8,477,149 B2 Page 2 U.S. PATENT DOCUMENTS 7,508,455 B2 3/2009 Liu et al. 2005/0212820 Al* 912005 Liu et al. ....................... 345/620 6,052,648 A * 412000 Burfeind et al. .................. 702/3 2007/0070226 Al * 3/2007 Matusik et al ................ 348/275 6,122,014 A * 912000 Panusopone et al. ......... 348/592 2007/0184422 Al* 8/2007 Takahashi ..................... 434/262 6,335,765 Bl * 1/2002 Daly et al. .................... 348/586 2008/02467 59 Al * 10/2008 Summers ...................... 345/420 6,437,782 Bl 8/2002 Pieragostini et al. 2009/0237 566 Al * 912009 Staker et al. 348/5 84 6,501,512 B2 12/2002 Nguyen et al. 2010/0111370 Al* 5/2010 Black et al. 382/111 6,807,296 B2 10/2004 Mishima 2010/0245387 Al* 9/2010 Bachelder et al. ............ 345/633 6,897,984 B2 512005 Warthen 2011/0025918 Al* 212011 Staker et al. 348/5 84 6,927,803 B2 8/2005 Toyama et al. 7,006,155 Bl 212006 Agarwala et al. * cited by examiner U.S. Patent Jul. 2, 2013 Sheet 1of6 US 8,477,149 B2 13 FOREGROUND -----.--__,__/__--"l>l- FOREGROUND ~11 VIDEO ,., SUPPRESSION I \ FORE 'I FOREGROUND FG KEY KEY 23 "'\ COMPOSITED GENERATOR VIDEO SUM BACKGROUND BG KEY KEY GENERATOR \I /17 BACKGROUND BACKGROUND VIDEO SUPPRESSION 14 FIG. 1 REFLECTED MONOCHROMATIC BLUE OR GREEN LIGHT SCREEN CAMERA LIGHT SOURCE FIG. 2 U.S. Patent Jul. 2, 2013 Sheet 2 of 6 US 8,477,149 B2 ------- I' ""- __..--- I ""- ------- ""- ------- I ------~ ------- •• ""- I I I' ""- ------ ------- ------- I I ""- 0.5 ------ I ""- I I I ""- I I I I __,,,. ------ +I ""- ""- I I ""- I I I ""- ~ I I ------ I ""- ""- I ------ ------- I 0.25 I ""- I ""- I I 'l ~ e 2 ""- ""- I I ""- I I ~ I ""- I ------- ------ ------- ""- 0 ------ ------- ""- I ""- ------- ------ ""- 0.5 ------ ""- ""- ""- '' \: 0.5 ""- FIG. 3a U.S. Patent Jul. 2, 2013 Sheet 3 of 6 US 8,477,149 B2 ~~~ e1 ~ I ~ I ~ ~ ~ ~ ~ }~ ~ I ~ 0.5 ~ I ~ I I - rout rin I I ~ ~ I I ~ ~~ ~ I ~ 1~~ I 0.25 I ~ ~I I '-1 ~ I I ~ I I I ~ ~~ ~ ~ ~ ~ ~ I ~ ~ ~ ~ 0 ~ >~ I ~ 0.5 ~ ~ ~ " 0.5 ~ FIG. 3b U.S. Patent Jul. 2, 2013 Sheet 4 of 6 US 8,477,149 B2 ------- I ------- ------- ------- I 0.5 ------- I I I I 0.25 0 - 0.5 0.5 0 FIG. 3c U.S. Patent Jul. 2, 2013 Sheet 5 of 6 US 8,477,149 B2 I Training I J1 Randomly sampling chroma color from several viewing angles and positions using a target camera to produce N triplet data points representing a data set of pixels J; Running principle components analysis (PCA) using the N triplet data points as input to calculate eigenvectors having nine values and three three-dimensional (30) vectors, eigenvalues having three values, and means having three values of the data set J; Constructing a whitening function from a mean vector (µ), column-major matrix of eigenvectors (E = [e 1 e 2 e 3 ]), and diagonal matrix of eigenvalues (A = diag p, 1A2 A3)) 1 2 comprising M(Ix) = A- / ET(Ix-µ); and J; j Output a triplet of a value. f FIG. 4a U.S. Patent Jul. 2, 2013 Sheet 6 of 6 US 8,477,149 B2 Classifying For all pixels, p, in a video frame: Calculate z=M(p), wherein z is a triplet representing standard deviations in the three independent dimensions of pixel p in chroma color distribution ,1, Receive a selection of a classifier function, C, and r compute o: =C( z), where o: is the alpha channel '-- I I of the pixel p I I I I I I I I Outputting the alpha matte I I I I *Sphere: Assume the data distribution is a 30 Gaussian. I I (a) calculate the 30 distance to the mean in the I transformed color space ( d = magnitude )z) ); I (b) using the distribution, find a lower and upper bound on single I I dimension z score ( d will be used with these upper /lower bounds); I ( c) calculate alpha by linearly interpolating d between I the lower and upper bound. I I *Cube: Assume each dimension of the data follows a Gaussian distribution. (a) calculate each ID distance to the mean in the transformed color space for each dimension (di =abs(z i )); (b) assume the worst z score ( d i) encloses the data distribution best, reduce dimensionality by taking the max distance across all 3 dimensions; ( c) using the distribution, find a lower and upper bound on single dimension z score; (d) calculate alpha by linearly interpolating dmax between the lower and upper bound. FIG. 4b US 8,477,149 B2 1 2 REAL-TIME CHROMAKEY MATTING USING spectrum, from a captured image to define a matte for com IMAGE STATISTICS positing multiple image sources. The problem is under-con strained and most commonly addressed through commercial CROSS-REFERENCE TO RELATED products that require tweaked parameters and specialized APPLICATIONS hardware for real-lime processing. In Mixed Reality (MR) applications using video see-through head-mounted display This application claims priority from U.S. provisional (HMD) hardware, the problem becomes more difficult since application Ser. No. 61/165,740, filed Apr. 1, 2009, and U.S. the poor camera quality within the HMDs, interactive pro provisional application Ser. No. 61/318,336, filed Mar. 28, cessing requirements of MR, and multiple camera sources 2010, the disclosures of which are incorporated herein by 10 demand a robust, fast, and affordable solution.