sign in sign up
<<
Home , White point

CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

000 054 001 055 002 056 003 Display Reshaping for Color Emulation and Balancing 057 004 058 005 059 006 060 007 061 008 062 009 Abstract 063 010 064 011 Emerging next generation digital light projectors are us- 065 012 ing multiple LED/laser sources instead of one lamp. 066 013 This results in a color gamut much larger than any exist- 067 014 ing display or capture device. Though advantageous in the- 068 015 ory, when used to display contents captured/processed at a 069 016 smaller gamut, a large gamut expansion results in -shift 070 017 artifacts. 071 018 We present a hardware-assisted 3D gamut reshaping 072 019 method that handles the gamut expansion in LED based 073 NTSC Traditional Single Source DLP 020 DLP displays by hierarchical temporal multiplexing of the Projectors (-2.9, -12.41) 074 HDTV (+7.7, +10.3) Multiple LED Source DLP 021 multiple primaries. This, in turn, results in a color emula- Projectors (+71, +66.6) 075 022 LCD Panels/Traditional Multiple Laser Source DLP 076 tion technique by which projectors with such large Singl le S ource LCD P roj ect ors PjtProjectors (+ 151, +96 .9) 023 can also achieve a standard color gamut and white point – (+11.5, +3.8) 077 024 the two most important color properties in terms of display Figure 2. Comparison of the different standard 2D color gamuts 078 025 quality, with an additional advantage of increased bright- with the gamuts provided by the LED or laser based projectors, 079 026 ness and dynamic range. The same method can also be used both in CIE xy and u’v’ space. The bold numbers indicate the 080 027 for color balancing across multiple projectors that are often percentage deviation in the area of the 2D gamut when compared 081 028 used to create large-scale high resolution displays. to the NTSC gamut, both in CIE xy and u’v’ space respectively 082 029 083 ple light sources, one for each primary, created from one 030 1. Introduction 084 031 or more light emitting diodes (LEDs) [8, 29, 10]. The pri- 085 maries are then switched ON and OFF or multiplexed tem- 032 The traditional digital light projection (DLP) technology 086 porally independent of each other (Figure1). 033 includes a white light bulb and a color wheel with differ- 087 034 ently colored filters. The filters are temporally multiplexed The LEDs in these projectors provide more saturated col- 088 035 at a high speed to selectively pass any one of the multiple ors than the color wheel resulting in a much larger color 089 036 primaries at any instant of time on to the digital micromir- gamut than any traditional projector and standard color 090 037 ror device (DMD) array [34]. The number of filters on the gamuts like NTSC, PAL, and even the most recent HDTV 091 038 color wheel can be three (R,G and B), four (R,G, B and (56% larger in the CIE u’v’ chart). In fact, 092 039 W) or more [30, 13, 24,1, 25,6, 15](Figure1). These are the emerging laser projectors, due to monochromatic pri- 093 040 wide band filters creating a gamut smaller than the stan- maries, promise to provide an even larger color gamut, cov- 094 041 dard industry-specified gamuts like NTSC, PAL and HDTV ering almost all the colors visible to the human eye (almost 095 042 (Figure2). Hence, media in one of these standard gamuts is double than that of the NTSC gamut in the CIE u’v’ space) 096 043 mapped to the smaller gamut of the display. [20]. The percentage increase/decrease of different display 097 044 gamuts when compared to the NTSC gamut both in the CIE 098 045 xy and u’v’ chromaticity charts is quantified in Figure2. 099 046 Though larger color gamut assures reproducibility of 100 047 a larger range of chrominance, this causes gamut expan- 101 048 sion creating several problems (e.g. hue-shifts, white- 102 049 point shift and non-optimal utilization of color resources) 103 050 Figure 1. Left to right: Light Path of a traditional DLP projector when displaying existing media generated in devices with 104 051 and a DLP projector with multiple LED sources (Blue filter is ON). a much smaller gamut(Section3). As a result, these up- 105 052 The projection industry has recently introduced projec- coming projectors cannot be used in applications using 106 053 tors where the color wheel is eliminated by using multi- projectors and cameras in a tightly coupled feedback loop 107

1 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

108 [18, 17, 21, 37, 19, 35, 28]. When the projected images are The XYZ coordinates of a color can be derived easily from 162 109 captured by the lower gamut cameras severe gamut clipping (I, x, y) using 163 110 artifacts occur. 164 111 In this paper, we present an algorithm to address this (X,Y,Z) = (xI, yI, I(1 − x − y)). (2) 165 112 gamut expansion. Unlike traditional single source projector 166 113 architecture where at any particular instance of time only Further, matching two colors, (I1, x1, y1) = (I2, x2, y2) 167 114 one of the primaries can be turned on, in projectors with assures that they also match in their XYZ coordinates i.e. 168 115 multiple LED sources more than one primary can be turned (X1,Y1,Z1) = (X2,Y2,Z2). Finally, the most important 169 116 on at the same time. Our method takes advantage of this point to note is that, for colors of similar chrominance, I 170 117 key property of simultaneous ON times to design a hard- scales proportionally to the luminance Y . Hence, in dis- 171 118 ware assisted scheme of hierarchical temporal multiplex- plays, for considering each primary or the grays, I and Y 172 119 ing of the LED primaries that can emulate a standard color are both scaled equally when the inputs are scaled. 173 120 gamut and white point without compromising other proper- It can be shown that in the CIE XYZ space, a ray through 174 121 ties like brightness, contrast and light efficacy, and enables the origin is the locus of colors with the same chromaticity 175 122 the following. coordinate (x, y) but different TTVs I. The chromaticity 176 123 1. Dynamic Color Emulation: Operability at standard color coordinates (x, y) is a 2D projection of these rays on the 177 124 gamut (like HDTV, NTSC and PAL) and white point (like X + Y + Z = d plane. The set of all chrominance visi- 178 125 D85 and D65) is very important for any display. Our ble to the human eye creates a horse-shoe shaped plot in the 179 126 method emulates many different color gamut and white xy space that represents different chrominance values phas- 180 127 point standards from the same set of LED primaries dy- ing out the I. This is called the chromaticity chart (Figure 181 128 namically as demanded by the application, just by chang- 2). The point (0.33, 0.33) in this chart indicates a perfect 182 129 ing the parameters of the temporal multiplexing (Section achromatic color with X = Y = Z. As the colors move 183 130 4). These parameters can be precomputed automatically away radially from this point towards the periphery of the 184 131 and then stored in the projector itself. horse-shoe shape, they change in saturation, while the hue 185 132 2. Robustness to Manufacturing Imprecision: The only way remains constant. 186 133 to achieve a desired color specification in a traditional sin- Finally, it can be shown with simple algebra, that adding 187 134 188 gle source projector is to control the color properties of their two colors, (I1, x1, y1) and (I2, x2, y2), result in a color 135 189 color filters via precision manufacturing. Since our method (I3, x3, y3) where I3 is the sum of the TTVs of the super- 136 can achieve the same standard color properties from LEDs imposing colors and chrominance is the weighted convex 190 137 that have a large variation in color, such strict control in combination of the chrominance of the superimposing col- 191 138 manufacturing can be avoided. This can make the technol- ors in the xy chromaticity chart, where the weights are given 192 139 ogy more flexible and cost effective. by the proportion of their TTVs. In other words, 193 140 194 3. Color Balancing in Multi-Projector Displays: Finally,   141 x1I1 + x2I2 y1I1 + y2I2 195 the same hierarchical scheme can also be used to achieve (I3, x3, y3) = I1 + I2, , . 142 color balancing across multiple projectors, common for I1 + I2 I1 + I2 196 143 building large-area high-resolution displays (Section 4.2). (3) 197 144 2. Notation This result can be generalized to n colors, where the 198 145 chrominance of the new color lies within the convex hull 199 146 We first briefly describe our color notation. Our algo- of the chrominance of the constituting n colors. Thus, in 200 147 rithm involves only color matching and does not deal with a projector with three primaries, the reproducible chromi- 201 148 color distances. Hence, all computations in our algorithm nance lies within the triangle spanned by the chrominance 202 149 are carried on in CIE XYZ space. However, when evaluat- of the three primaries (Figure2). This is called the 2D color 203 150 ing the display quality, we use a perceptually uniform color gamut or simply the color gamut. The chrominance of the 204 151 space, like CIELAB or CIELUV space. white created by full intensity primaries superimposed from 205 152 Let (X,Y,Z) be the 3D coordinates of a color in the CIE three channels is called the white point. This depends on the 206 153 XYZ space, called the tristimulus values. In our algorithmic proportion of the TTVs of the three primaries and need not 207 154 computations, total tristimulus value (TTV) X +Y +Z (the be the perfect white, (0.33, 0.33). The brightness of a de- 208 155 indicator of the total energy of the spectrum) plays an im- vice is defined by the luminance Y of full intensity white, 209 156 portant role. Hence, we specify the (X,Y,Z) color alter- and is proportional to its TTV. Note that the white point 210 157 natively by its TTV I = X + Y + Z, and its chromaticity and 2D color gamut specifies the chrominance capabilities 211 158 coordinates (the indicator of its chrominance), (x, y), de- while brightness specifies the dynamic range capabilities. If 212 159 fined as the brightness of black is constant, a higher brightness indi- 213 160 X Y cates a higher dynamic range. When considering brightness 214 (x, y) = ( , ). (1) 161 X + Y + Z X + Y + Z and chrominance together, the XYZ values of the primaries 215

2 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

216 gamut, display. Hence, in any projector camera applica- 270 217 tion [18, 17, 21, 37, 19, 35, 28, 22] when images of the 271 218 gamut expanded media on an LED projector is captured by a 272 219 lower gamut camera, gamut clipping results in severe color 273 220 blotches (Figure3). 274 221 Gamut expansion is prevalent in a much smaller scale 275 222 when moving from the smaller color gamut of a printer to a 276 223 larger gamut of a monitor, but not as pronounced as in the 277 224 context of projection-based displays, especially when using 278 225 multiple primaries [15, 20]. However, a similar scenario 279 226 is prevalent currently in the context of dynamic range of 280 227 displays [27] and has led to development of inverse tone 281 228 mapping methods that map LDR content to HDR displays 282 229 [23,2]. Gamut being a 2D/3D entity, as opposed to 1D 283 230 dynamic range, makes gamut expansion a more complex 284 231 problem. In this section, we briefly visit the relevant works 285 232 Figure 3. Effects of No Color Management – Row 1: Images in in this direction. 286 233 NTSC color gamut; Row 2: The representation of nor- 287 234 malized hue shift in the perceptually uniform CIELAB space due 288 235 to gamut expansion in LED projectors (Brighter grays indicate 289 236 more hue-shift) – the normalization is with respect to the maxi- 290 mum hue shift seen in each image; the max and mean hue shift 237 291 in the three images from left to right are (52.5, 32.8), (55.2, 39.1) 238 292 and (54.3, 41.8) respectively; Row 3: The images displayed on the 239 LED projector with a larger 2D color gamut is then recaptured by 293 240 a standard NTSC camera. Note that due to shifting of colors out- 294 241 side of the NTSC space during display, the recaptured image loses 295 242 many of the colors due to gamut clipping. 296 243 297 244 span a 3D parellelopipped in the CIE XYZ space. This con- 298 stitutes the range of all different colors (both chrominance 245 Figure 4. Comparisons of the 3D gamuts in CIE XYZ space of a 299 246 and brightness) that can be reproduced by the device, called traditional projector (green) and a LED projector assuming their 300 247 the 3D color gamut. primaries have the same TTV (red), and 1.5 times the TTV (blue). 301 248 3. Gamut Expansion and Related Work Gamut Clipping: Standard gamut clipping techniques 302 249 [33,7, 14,5] are currently used when converting between 303 250 A smaller gamut media displayed on a larger gamut dis- gamuts of similar shape and volume. The source input 304 251 play without applying any color management techniques 0 ≤ (r, g, b) ≤ 1 is mapped to a target color (r0, g0, b0) using 305 252 (i.e., no modification of content) can show visible color in- a linear transformation M, i,e. (r0, g0, b0)T = M(r, g, b)T . 306 253 coherence (jarring hue-shifts) due to the increase in the per- The resulting (r0, g0, b0), if outside the target 3D gamut is 307 254 ceptual distance between two perceptually coherent colors. clipped to the boundary of the 3D gamut. However, this 308 255 Figure3 visualizes this hue-shift at every pixel in percep- can have some adverse effects on gamut expansion. First, 309 256 tually uniform CIELAB space as a gray scale image nor- a larger 2D color gamut of the LED projector does not 310 257 malized with respect to its maximum hue-shift in the im- necessarily indicate a larger 3D color gamut. If the lumi- 311 258 age (since this cannot be captured or printed in a device nance (and hence the TTVs) of the primaries are similar, 312 259 with a smaller gamut than the LED projector). Since a hue- the standard NTSC 3D gamuts may be significantly differ- 313 260 shift of 3-5 in CIELAB space is easily visible [38], the hue- ent in shape and not contained within the LED projector’s 314 261 shifts resulting from the gamut expansion (between 30-55) 3D gamut (Figure4). Hence, clipping of transformed col- 315 262 is very significant. Note that the color shifts are predomi- ors still maps multiple source colors to a single target color 316 263 nantly in the red-yellow (fall image) and green-yellow (gar- resulting in color blotches (Figure3 bottom row). 317 264 den image) region of the chromaticity chart where most of Gamut Extension: Instead of clipping only the out-of- 318 265 the gamut expansion occurs (Figure2). It also causes a gamut colors, other techniques aim at moving all differ- 319 266 significant white point shift (e.g. frog image). A NTSC ent colors in an optimized fashion from the smaller source 320 267 gamut with standard D65 white (chromaticity coordinate gamut to the larger target gamut. [12, 26] apply hue- 321 268 of (0.3127, 0.3290)) shifts considerably to the greenish- preserving color extrapolation by changing only the bright- 322 269 blue white region (0.272, 0.369) when used for the larger ness (and hence TTVs) and saturation, and [15,9] apply 323

3 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

324 complex non-linear optimizations on perceptually uniform the multiple primaries in a controlled, deterministic (non- 378 325 3D color spaces constraining the movement of colors within iterative) manner so that any specified 2D color gamut and 379 326 an acceptable distance. However, since the increase in white point that lies within the gamut can be achieved. 380 327 gamut volume in LED projectors is much more drastic than 4.1. Color Emulation for a Single Projector 381 328 what is expected in these methods, they result in perceivable 382 Our method takes two sets of inputs – (a) the target (de- 329 hue shift [15] and may not utilize the entire gamut. 383 sired) color specifications, i.e., of the pri- 330 Comparison of Proposed Method: Our method is com- 384 331 maries and white point; (b) the chromaticity coordinates 385 plementary to all the above methods. In contrast to com- and the TTV, (I, x, y), of the LED primaries at full inten- 332 plex gamut extension techniques [12,5, 14, 15, 26,9], the 386 333 sity, measured by a radiometer. The output is the pulse 387 biggest advantage of our method lies in its sheer simplic- width modulated ON-times for superimposing the primaries 334 ity - both in concept and in computation. Unlike existing 388 335 to achieve the target specifications. Note that this is appli- 389 gamut extension techniques our method can be directly im- cable even to systems with more than three primaries. 336 plemented using the existing DLP hardware. Further, our 390 337 Let the measured color properties of the LED primary l, 391 method can exactly emulate a predefined 2D gamut or white l ∈ {R, G, B}, be (I ,C ) where C = (x , y ) is the chro- 338 point, without working within the confines of the 3D gamut l l l l l 392 339 maticity coordinate of l. Let the target 2D color gamut be 393 provided by an optimization. Finally, the hierarchical na- 0 0 0 340 defined by the new primaries R , G and B whose chro- 394 ture of our method (Section 4.1.3) allows a larger flexibility 0 0 0 0 maticity coordinates are C 0 , l ∈ {R ,G ,B } and the 341 in terms of the emulated properties and extension to multi- l 395 white point chrominance is C 0 = (x 0 , y 0 ). 342 projector displays (Section 4.2). W W W 396 343 The output of the different intensity levels in LED-based 397 344 DLP projectors are achieved via temporal multiplexing of 398 345 the ON times of each LED primaries, R, G and B [11]. 399 346 This forms the Level 0 of our hierarchical scheme. Each 400 347 of the successive levels run two methods: (a) temporal- 401 348 modulation that determines the relative ON times of the 402 349 different LEDs which will be turned ON simultaneously to 403 350 achieve the target specification; (b) TTV-computation that 404 351 determines the TTV of the new primaries thus formed by the 405 352 temporal modulation step. Figure6 illustrates the method. 406 353 4.1.1 Gamut Emulation in Level 1 407 354 408 355 The goal of this step is to superimpose the colors from 409 356 more than one LED to transform the larger 2D gamut to 410 357 Figure 5. The 3D NTSC gamut (green), 3D gamut of the LED the smaller standard 2D gamut like NTSC (Figure2). 411 projector before our method is applied (red) and the reshaped 358 Temporal Modulation: We control the ON times of the 412 3D gamut of the projector after our method is applied (blue), in 359 three LEDs to realize the proportions of the LED primaries 413 CIEXYZ space. The projection of the three basis vectors that span that achieve the target primaries. tij, denotes the ON-time 360 the 3D gamut on a X+Y+Z=d plane is the 2D color gamut. Note 414 361 for LED primary i, i ∈ {R, G, B}, required to create the 415 that after applying our method, the basis vectors of the reshaped 0 0 0 target primary j, j ∈ {R ,G ,B }. For e.g. t 0 denotes 362 3D gamut is coincident with the NTSC gamut but is bigger. This GR 416 the ON-time of the green LED primary to create the target 363 assures that the extra 3D volume is used to increase the brightness 417 364 (and hence the dynamic range), rather than the 2D gamut. red primary. 418 First, we compute a 3 × 3 matrix that transforms the tri- 365 4. Hardware Assisted Gamut Reshaping 419 366 angle CRCGCB to CR0 CG0 CB0 , given by 420 367 Most of the extra volume of the 3D gamut of LED pro-       421 CR0 dR0 eR0 fR0 CR 368 jectors (Figure4) is used to reproduce a larger range of 422  CG0  =  dG0 eG0 fG0   CG  (4) 369 chrominance, while the range of reproducible brightness, 423 CB0 dB0 eB0 fB0 CB 370 given by the proportional TTVs, is still similar. We present 424 371 a content-independent hardware-assisted algorithm that re- where dl0 , el0 , fl0 denote the proportion of R,G and B 425 372 shapes this 3D gamut using hierarchical temporal multi- (barycentric coodinates) required to create the target pri- 426 0 0 0 0 0 373 plexing, so that most of this extra volume is instead uti- maries l , l ∈ {R ,G ,B }. Since fl0 = 1 − dl0 − el0 , there 427 374 lized to increase the brightness, while the 2D color gamut is are only six unknowns which are provided by the known Cl 428 375 matched to a standard 2D color gamut (Figure5). and Cl0 . 429 376 Our algorithm uses this property of simultaneous ON For IR = IG = IB, dl0 , el0 and fl0 can be directly as- 430 377 time of the primaries to design a scheme to superimpose signed to tRl0 , tGl0 and tBl0 respectively. However, since 431

4 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

432 Primaries White Point 486 1 unit 433 1 unit 1 unit 1 unit = 11mS 487 434 488 R Level 0 435 G 489 436 B 490 437 R 491 tRR’ tRG’ tRB’ 438 G Gamut 492 tGR’ tGG’ tGB’ Emulation 439 B 493 tBR’ tBG’ tBB’ (Level 1) 440 R’ G’ B’ 494 441 495 442 R White point 496 G Emulation 443 B 497 (Level 2) t t 444 tR’ G’ B’ 498 R’’ G’’ B’’ 445 499 R Color 446 G 500 B Balancing 447 With Other 501 448 Projectors 502 ktR’ ktG’ ktB’ 449 (Level 3) 503 RM GM BM 450 504 Figure 6. This shows the 33mS time interval for displaying a single frame of a 30fps video. The colored bars show how much time each of 451 505 the primaries are ON in each interval for each level of the hierarchy. The circles on the right hand side show how the color properties of 452 the display changes with each level. Note that this is just an illustration of the computation and does not correspond to any real data. 506 453 507 454 IR 6= IG 6= IB in reality, the ON times tRl0 , tGl0 and tBl0 other primaries) leading to a brighter projector. 508 455 required to achieve the target chrominance Cl0 must take 509 456 this into consideration. For example, if IR < IG, then IR 510 457 should be kept ON longer to provide the desired TTV pro- 511 458 portional to dl0 . Thus, the timing is a function of both the 512 459 chrominance and TTV of the source LED primaries. For 513 460 example, the tlR0 , l ∈ {R, G, B} is given by 514 461 515 IR + IG + IB 462 t 0 = d 0 × (5) 516 RR R I 463 R 517 IR + IG + IB 464 tGR0 = eR0 × (6) 518 465 IG 519 466 IR + IG + IB 520 tBR0 = fR0 × (7) 467 IB 521 468 522 Since, R0 is formed by the superposition of all three of R,G Figure 7. The 3D gamut in CIEXYZ space at Level 0 (red), Level 469 1 (magenta) and Level 2 (blue) of our method. 523 and B, these timings are normalized by the maximum of 470 524 t 0 , t 0 and t 0 . For simplicity, we retain the same no- 471 RR GR BR 4.1.2 White Point Emulation in Level 2 525 tation for the normalized timings. Similar computations are 472 526 performed to compute the timings for G0 and B0. For gen- 473 This step realizes the target white point without affecting 527 erating C 0 , t 0 will usually be much larger than t 0 and 474 R RR GR the new primaries achieved in the previous level using the 528 t 0 since C 0 is much closer to C than to C and C . 475 BR R R G B same steps of temporal modulation and TTV computation 529 Hence, for C 0 , t 0 = 1 after normalization. Similarly, 0 0 0 476 R RR but with the new primaries R , G and B . 530 after normalization, t 0 = 1 and t 0 = 1 for C 0 and 477 GG BB G Temporal Modulation: We modify the contributing pro- 531 C 0 respectively. 478 B portions of the new primaries, R0, G0 and B0, in such a 532 TTV Computation: The temporal modulation creates new 479 manner that the chromaticity of target white point, C 0 , 533 primaries R0, G0 and B0 whose TTV we compute next. The W 480 0 0 0 0 is matched. Let this target proportion be pR0 : pG0 : pB0 534 TTV of the new primary l ∈ {R ,G ,B }, is Il0 and is 481 where p 0 = 1 − p 0 − p 0 . The equation is given by 535 given by B R G 482 536 Il0 = tRl0 IR + tGl0 IG + tBl0 IB (8) 483 pR0 (xR0 , yR0 )+pG0 (xG0 , yG0 )+pB0 (xB0 , yB0 ) = (xW 0 , yW 0 ). 537 484 Since tRl0 = 1, Il0 > Il. Thus, the new primaries are (9) 538 485 brighter (due to superimposition of additional light from Thus, the ON times for the new primaries to achieve the 539

5 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

540 Desired Gamut/ Increase Decrease in Volume 594 541 White Point in CIE Y of 3D Gamut 595 542 Lvl 1 Lvl 2 Lvl 1 Lvl 2 596 543 HDTV/D65 20.8% 17.1% 3.0% 5.5% 597 544 (USA) 598 545 599 HDTV/D85 20.8% 19.1% 3.0% 11.1% 546 600 (Japan/Korea) 547 601 NTSC/D65 23.4% 20.1% 4.4% 7.2% 548 602 (USA) 549 603 NTSC/D85 23.1% 21.8% 4.4% 13.8% 550 604 (Japan/Korea) 551 Figure 9. Please zoom in to see our curved screen display made of 605 PAL/D65 20.1% 15.9% 2.8% 5.5% 552 four projectors (real system, not simulated) running our color bal- 606 (Europe/India) 553 ancing algorithm. Top: This shows the before (left) and af- 607 ter (right) our color emulation matching to an NTSC gamut. Note 554 608 Table 1. Statistics of how the brightness and volume of the 3D that the brightness balancing across projectors described in Sec- 555 609 gamut of the projector changes in Level 1 and Level 2 from its tion 4.2 is still not applied. Hence, intensity variation and seams 556 original state in Level 0. are still visible. Middle: This shows a regular content (Venice) be- 610 557 fore (left) and after (right) our color emulation for each projector to 611 558 match the NTSC gamut, followed by our TTV (and hence bright- 612 559 ness) balancing across projectors and finally removing spatial vari- 613 560 ation in TTV (and hence in brightness) using existing camera- 614 561 based calibration methods. Bottom: In this zoomed in views, com- 615 562 Figure 8. This figure demonstrates the white point emulation. The pare the colors of the buildings near the text Hotel Marconi to 616 563 original non-standard reddish white (0.35,0.325) in CIE xy space see the effect of our color balancing. 617 564 (left) has been changed to a neutral D65 white (middle) and a 618 Table1 compares the volume of the 3D gamut and the 565 bluish D85 white (right) using our white emulation. 619 TTV of the white (I 0 ), finally achieved by the color em- 566 target white point is W 620 ulation method. Note that since for the equi-chrominance 567 P 621 0 0 0 0 Il0 grays of a display, I and luminance Y are scaled similarly 568 l ∈{R ,G ,B } 622 tl0 = pl0 × (10) with the input (Section2), I 0 provides a direct measure 569 Il0 W 623 of the increase in the display brightness. However, the loss 570 In this step, the normalization of the timings is different. 624 in the volume is very small when compared to the gain in 571 Unlike Level 1 where the LED primaries R, G and B were 625 brightness, which can be as large as 25%. 572 superimposed, in Level 2 the primaries R0, G0 and B0 are 626 Hierarchical Nature of the Scheme: Our method is hi- 573 lighted sequentially for a total of 3 units of time. Hence, tl0 627 574 tR0 +tG0 +tB0 erarchical in nature where each level of the hierarchy mod- 628 should be normalized by 3 . Here also, we retain 575 the same notation for the normalized timings for simplicity. ifies the primaries which are then used as new primaries in 629 576 TTV Computation: Hence, the TTV of the new pri- the subsequent level. This provides a few nice properties 630 to our algorithm. (a) Level Independency: In each step of 577 maries is given by tl0 Il0 . The TTV of the white, IW 0 , 631 the hierarchy, only one property of the display is modified 578 is the sum of the TTVs of these new primaries IW 0 = 632 579 P independently (e.g. 2D color gamut in Level 1 and white 633 l0∈{R0,G0,B0} tl0 Il0 . IW 0 is proportional to the luminance 580 of the white which is the measure of the display bright- point in Level 2). (b) Preservation of Lower Level Prop- 634 erties: A color property standardized at a particular level 581 ness. In Section 4.1.3, we show that IW 0 is greater than 635 of the hierarchy is preserved through the subsequent levels. 582 IW = IR + IG + IB in the original projector. 636 583 (e.g. changing white point in Level 2 does not change the 637 4.1.3 Discussion 0 0 0 584 new primaries R , G and B ). (c) Module Invariance: The 638 585 This section offers some useful insights to the transforma- same computations are used in each level of the hierarchy 639 586 tions achieved by our method. but with different inputs to impact different properties. 640 587 Reshaping of the 3D Gamut: Figure7 shows how the Flexibility of the Scheme: Note that we can precom- 641 588 3D gamut reshaping happens across different levels of our pute timings to achieve multiple different specifications (for 642 589 hierarchical scheme. The 3D gamut reduces in volume from e.g. HDTV and D65 white or NTSC and D85 white) and 643 590 Level 1 to 2 due to the constraints imposed by a specific store them in a look-up-table (LUT) in the projector itself. 644 591 target white point. However, the final 3D gamut is always The application software can then use this to create differ- 645 592 bigger than the standard industry specfied 3D color gamut ent color properties as the desired white point or gamut used 646 593 but smaller than the original LED 3D gamut. in devices changes from country to country (from NTSC 647

6 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

648 tors in two steps. (a) First, we use our color emulation 702 649 method to match all the different projector to the same stan- 703 650 dard color gamut and white point (Section4). (b) Next, to 704 651 balance the still varying brightness we add an extra level to 705 652 our hierarchical method (Section 4.1). 706 653 Let us assume n projectors, with projector i having the 707 654 708 TTV for white I 0 following the first two levels. We seek Wi 655 to match the TTV (and hence brightness) across multiple 709 656 710 projectors, such that for any i, j ∈ 0, . . . n − 1, I 0 = I 0 . Wi Wj 657 First, we choose the minimum TTV of all projectors as the 711 658 712 target TTV IM = mini∈0,...n−1 IWi . Next the ON period 659 of each of the new primaries of the projector i are scaled 713 660 Figure 10. This shows our planar display made of 16 projectors 714 by k = IM matching the TTV (and hence brightness) of (real system, not simulated) after color balancing using the follow- i I 0 661 W 715 ing three steps in succession: (a) color emulation of each projector i 662 all projectors to IM . The new primaries thus formed are 716 to NTSC gamut (Section4); (b) color balancing across projectors 663 denoted by RM , GM and BM . However, since the relative 717 (Section 4.2); (c) removal of spatial vignetting effects by existing proportions of the primaries are not changed, the 2D color 664 camera-based registration methods. 718 665 gamut and the white point are unaffected by this step. 719 666 in US to PAL in Europe/India, from D65 in US to D85 in Our method can only achieve color and brightness 720 667 Japan/Korea). balancing across projectors, but cannot handle the intra- 721 668 4.1.4 Implementation and Results projector spatial variation in brightness (commonly called 722 669 vignetting effect). For this, existing camera-based bright- 723 We tested our color emulation on over 20 different pro- 670 ness calibration methods [16] can be used following our 724 jectors to realize NTSC gamut and D65 color point. The 671 color balancing scheme. Since our hierarchical temporal 725 time multiplexing was realized using the DLP chip hard- 672 multiplexing method has already modified the larger projec- 726 ware. Figure8 shows the results of changing the white point 673 tor gamut to be close to that of the camera, using a camera 727 from the existing white point (shifted towards red) to a D65 674 in the feedback loop no longer poses a problem. 728 (neutral) and D85 (shifted towards blue) white. Since, the 675 729 saturated primaries of the projectors cannot be reproduced 4.2.1 Implementation and Results 676 730 either in print or in existing displays, we provide statistical 677 We have implemented our color balancing technique on two 731 data on the accuracy of the gamut mapping. Our red, green 678 displays – (a) a curved display made of four projectors (Fig- 732 and blue LEDs had mean of 617nm, 679 ure9); and (b) a planar display made of 16 projectors (Fig- 733 520nm green and 464nm with a variation about 16nm, 8nm 680 ure 10). The whites on the calibrated display in the top 734 and 6nm respectively. We achieved an NTSC gamut for all 681 row of Figure9 is only after application of our color emu- 735 the projectors up to an absolute error of 10−4 in the CIE xy 682 lation method, as described in Section4. Hence, the bright- 736 chromaticity chart. Since we are dealing with color match- 683 ness variation across projectors and the spatial vignetting is 737 ing and not dealing with perceptual distances, use of CIE xy 684 still evident. All the other images of calibrated displays are 738 space is justified. All measurements (before and after color 685 achieved by applying three steps: (a) color emulation, as in 739 emulation) were from the same spot on the projector using 686 Section4; (b) brightness balancing as in Section 4.2; and 740 a Photo Research 705 Spectrascan spectroradiometer. 687 (c) removal of spatial vignetting by applying the methods 741 4.2. Color Balancing Multiple Projectors 688 in [16]. Note that these three steps make both our four and 742 689 The advent of low-cost LED projectors bring in the sixteen projector displays perceptually seamless. 743 690 potential of building very high-resolution tiled projection- 5. Conclusion 744 691 based displays that are both portable and affordable [22,4, 745 692 3]. Currently, color balancing across multiple projectors is In conclusion, we demonstrated that the larger gamut of 746 693 achieved via software in two steps: (a) first, a common 3D the current LED projectors lead to several hue-shift arti- 747 694 color gamut contained within the gamut of all the projectors facts and sub-optimal utilization of color resources when 748 695 is computed; (b) next, a linear [31, 32] or piecewise linear displaying existing lower gamut media due to gamut ex- 749 696 [36] transformation is used to convert the gamuts of all the pansion. We presented a content-independent hardware- 750 697 projectors to the common gamut. For LED projectors such assisted method that reshapes the 3D gamut to match a stan- 751 698 a gamut mapping leads to degradation in image brightness dard 2D color gamut while directing the extra volume to- 752 699 and contrast. wards achieving a higher dynamic range. This method can 753 700 We can use our hierarchical temporal multiplexing be used for color emulation of single projectors and also for 754 701 scheme to achieve color balancing across multiple projec- color balancing multiple projectors. 755

7 CVPR CVPR #**** #**** CVPR 2010 Submission #****. CONFIDENTIAL REVIEW COPY. DO NOT DISTRIBUTE.

756 As is evident, in the future it is critical to explore ap- [18] S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar. 810 757 proaches to develop a sensor with large gamut so that the Fast separation of direct and global components of a scene 811 758 display resources can be utilized to its fullest. One can also using high frequency illumination. ACM TOG, 25(3), 2003. 812 759 explore better content-dependent approaches in the future 2,3 813 760 that can handle very large gamut expansions so that the [19] S. K. Nayar, H. Peri, M. D. Grossberg, and P. N. Belhumeur. 814 761 larger chrominance gamut offered by these emerging dis- A Projection System with Radiometric Compensation for 815 Screen Imperfections. IEEE PROCAMS, 2003.2,3 762 plays are optimally utilized. 816 [20] G. Niven and A. Mooradian. Low cost lasers and laser arrays 763 References 817 764 for projection displays. pages 1904–1907, 2006.1,3 818 765 [1] T. Ajito, T. Obi, M. Yamaguchi, and N. Ohyama. Expanding [21] C. Pinhanez, M. Podlaseck, R. Kjeldsen, A. Levas, G. Pin- 819 766 color gamut reproduced by six-primary projection display. gali, and N. Sukaviriya. Ubiquitous interactive displays in a 820 retail environment. ACM SIGGRAPH Sketches, 2003.2,3 767 Proceedings of SPIE, 3954:130–137, 2000.1 821 [22] R. Raskar, J. van Baar, P. Beardsley, T. Willwacher, S. Rao, 768 [2] F. Banterle, P. Ledda, K. Debattista, and A. Chalmers. In- 822 verse tone mapping. pages 349–356, 2006.3 and C. Forlines. ilamps: Geometrically aware and self- 769 configuring projectors. ACM TOG, 22(3), 2003.3,7 823 [3] E. S. Bhasker, R. Juang, and A. Majumder. Registration tech- 770 [23] A. G. Rempel, M. Trentacoste, H. Seetzen, H. D. Young, 824 771 niques for using imperfect and partially calibrated devices in 825 planar multi-projector displays. IEEE TVCG, 13(5), 2007.7 W. Heidrich, L. Whitehead, and G. Ward. Ldr2hdr: on-the- 772 fly reverse tone mapping of legacy video and photographs. 826 [4] E. S. Bhasker, P. Sinha, and A. Majumder. Asynchronous 773 ACM TOG, 26, 2007.3 827 distributed calibration for scalable reconfigurable multi- 774 [24] J. E. Roddy, R. J. Zolla, N. A. Blish, and L. S. Horvath. Six 828 projector displays. IEEE TVCG, 12(5):1101–1108, 2006.7 color display apparatus having increased color gamut. US 775 [5] G. Braun. A Paradigm for Color Gamut Mapping of Pictorial 829 Patent 6769772, 2004.1 776 Images. PhD thesis, RIT, Rochester, NY, 1999.3,4 830 [25] S. Roth, I. Ben-David, M. Ben-Chorin, D. Eliav, and O. Ben- 777 [6] D. Eliav, S. Roth, and M. B. Chorin. Application driven 831 David. Wide gamut, high brightness multiple primaries sin- 778 design of multi-primary displays. Proc. of IS&T/SID’s 14th 832 gle panel projection displays. SID Digest, 2003.1 779 Color Imaging Conference, 2006.1 833 [26] J. J. Sara. The Automated Reproduction of Pictures with Non- 780 [7] E. J. Giorgianni and T. E. Madden. Digital Color Manage- 834 reproducible Colors. PhD thesis, MIT, 1984.4 781 ment : Encoding Solutions. Addison Wesley, 1998.3 835 [27] H. Seetzen, W. Heidrich, W. Stuerzlinger, G. Ward, 782 [8] G. Harbers and C. Hoelen. High performance lcd backlight- 836 L. Whitehead, M. Trentacoste, A. Ghosh, and A. Vorozcovs. ing using high intensity red, green and blue light emitting 783 High dynamic range display systems. ACM TOG, 23(3), 837 diodes. SID Digest, pages 702–705, 2001.1 784 2004.3 838 [9] T. Hirokawa, M. Inui, T. Morioka, and Y. Azuma. A 785 [28] P. Sen, B. Chen, G. Garg, S. R. Marschner, M. Horowitz, 839 psychophysical evaluation of a gamut expansion algorithm 786 M. Levoy, and H. P. A. Lensch. Dual photography. ACM 840 based on chroma mapping ii: Expansion within object color 787 Transactions on Graphics, 24, 2005.2,3 841 data bases. NIP23, pages 175–179, 2007.4 788 [29] M. Shindoh. Projector optics and projector with light source 842 [10] R. L. Holman and A. Cox. High-density illumination system. of leds. US Patent 7101049, 2006.1 789 US Patent 7210806, 2007.1 843 [30] P.-L. Song. Projector apparatus. US Patent 6830343, 2004. 790 [11] L. J. Hornbeck. Field updated deformable mirror device. US 844 1 791 Patent 5280277, 1994.4 845 [31] M. C. Stone. Color balancing experimental projection dis- 792 [12] T. Hoshino. A preferred color reproduction method for the 846 plays. 9th IS&T/SID Color Imaging Conference, 2001a.7 793 hdtv digital still image system. pages 27–32, 1991.4 847 [32] M. C. Stone. Color and brightness appearance issues in tiled 794 [13] C. Joubert, C. Puech, B. Loiseaux, and J.-P. Huignard. Video 848 displays. IEEE CG&A, 2001b.7 795 image projector with improve luminous efficiency. US Patent 849 [33] M. C. Stone. A Field Guide to Digital Color. A.K. Peters, 5526063, 1996.1 796 2003.3 850 [14] B. H. Kang, M. S. Cho, J. Morovic, and M. R. Luo. Gamut 797 [34] E. H. Stupp and M. S. Brennesholtz. Projection Displays. 851 extension development based on observer experimental data. 798 John Wiley and Sons Ltd., 1999.1 852 IS&T/SID 9th Color Imaging Conference, pages 158–162, 799 [35] R. Sukthankar, R. Stockton, and M. Mullin. Smarter presen- 853 2001.3,4 800 tations: Exploiting homography in cameraprojector systems. 854 [15] M. C. Kim, Y. C. Shin, Y. R. Song, S. J. Lee, and I. D. Kim. 801 ICCV, 2001.2,3 855 Wide gamut multiprimary display for hdtv. Second Euro- 802 [36] G. Wallace, H. Chen, and K. Li. Color gamut matching for 856 pean Conference on Color in Graphics, Imaging and Vision, 803 tiled display walls. IPT Workshop, 2003.7 857 Aachen, Germany, page 248253, April 2004.1,3,4 [37] R. Yang, A. Majumder, and M. Brown. Camera based 804 [16] A. Majumder and R. Stevens. Perceptual photometric seam- 858 calibration techniques for seamless multi-projector displays. 805 lessness in tiled projection-based displays. ACM Transac- 859 IEEE TVCG, 11(2), March-April 2005.2,3 806 tions on Graphics, 24(1), January 2005.7 860 [38] Z. Zalevsky and A. Goldman. Spectral modeling and im- 807 [17] V. Masselus, P. Peers, P. Dutre, and Y. D. Willems. Relight- 861 provement of target detection in a visible cluttered environ- 808 ing with 4d incident light fields. ACM SIGGRAPH, 2003.2, 862 ment. Optical Engineering, 41:1358–1364, 2002.3 809 3 863

8