Depth and Transparency

Computer Graphics CMU 15-462/15-662 Today: Wrap up the rasterization pipeline! Remember our goal: • Start with INPUTS (triangles) –possibly w/ other data (e.g., colors or texture coordinates) • Apply a series of transformations: STAGES of pipeline • Produce OUTPUT (fnal image)

INPUT RASTERIZATION OUTPUT (TRIANGLES) PIPELINE (BITMAP IMAGE)

VERTICES A: ( 1, 1, 1 ) E: ( 1, 1,-1 ) B: (-1, 1, 1 ) F: (-1, 1,-1 ) C: ( 1,-1, 1 ) G: ( 1,-1,-1 ) D: (-1,-1, 1 ) H: (-1,-1,-1 ) TRIANGLES EHF, GFH, FGB, CBG, GHC, DCH, ABD, CDB, HED, ADE, EFA, BAF

CMU 15-462/662 What we know how to do so far…

(w, h) y z

x (0, 0) position objects in the world project objects onto the screen sample triangle coverage (3D transformations) (perspective projection) (rasterization)

???

put samples into frame buffer sample texture maps interpolate vertex attributes (depth & alpha) (fltering, mipmapping) (barycentric coodinates)

CMU 15-462/662 Occlusion

CMU 15-462/662 Occlusion: which triangle is visible at each covered sample point?

Opaque Triangles 50% transparent triangles

CMU 15-462/662 Sampling Depth Assume we have a triangle given by: – the projected 2D coordinates (xi,yi) of each vertex – the “depth” di of each vertex (i.e., distance from the viewer) screen

Q: How do we compute the depth d at a given sample point (x,y)? A: Interpolate it using barycentric coordinates (just like any other attribute that varies linearly over the triangle)

CMU 15-462/662 The depth-buffer (Z-buffer)

For each coverage sample point, depth-buffer stores the depth of the closest triangle seen so far.

Initial state of depth buffer before rendering any triangles (all samples store farthest distance)

Grayscale value of sample point used to indicate distance Black = small distance White = large distance (“infnity”)

CMU 15-462/662 Depth buffer example

CMU 15-462/662 Example: rendering three opaque triangles

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

Processing yellow triangle: Grayscale value of sample point depth = 0.5 used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

After processing yellow triangle: Grayscale value of sample point used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

Processing blue triangle: Grayscale value of sample point depth = 0.75 used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

After processing blue triangle: Grayscale value of sample point used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

Processing red triangle: Grayscale value of sample point depth = 0.25 used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth-buffer (Z-buffer)

After processing red triangle: Grayscale value of sample point used to indicate distance White = large distance Black = small distance Red = sample passed depth test

Color buffer contents Depth buffer contents

CMU 15-462/662 Occlusion using the depth buffer bool pass_depth_test(d1, d2) { return d1 < d2; } depth_test(tri_d, tri_color, x, y) {

if (pass_depth_test(tri_d, zbuffer[x][y]) {

// triangle is closest object seen so far at this // sample point. Update depth and color buffers.

zbuffer[x][y] = tri_d; // update zbuffer color[x][y] = tri_color; // update color buffer } }

CMU 15-462/662 Does depth-buffer algorithm handle interpenetrating surfaces? Of course! Occlusion test is based on depth of triangles at a given sample point. The relative depth of triangles may be different at different sample points.

Green triangle in front of yellow triangle

Yellow triangle in front of green triangle

CMU 15-462/662 Does depth-buffer algorithm handle interpenetrating surfaces? Of course! Occlusion test is based on depth of triangles at a given sample point. The relative depth of triangles may be different at different sample points.

CMU 15-462/662 Does depth buffer work with super sampling? Of course! Occlusion test is per sample, not per pixel!

This example: green triangle occludes yellow triangle

CMU 15-462/662 Color buffer contents

CMU 15-462/662 Color buffer contents (4 samples per pixel)

CMU 15-462/662 Final resampled result

Note anti-aliasing of edge due to fltering of green and yellow samples.

CMU 15-462/662 Summary: occlusion using a depth buffer ▪ Store one depth value per coverage sample (not per pixel!) ▪ Constant space per sample - Implication: constant space for depth buffer ▪ Constant time occlusion test per covered sample - Read-modify write of depth buffer if “pass” depth test - Just a read if “fail” ▪ Not specifc to triangles: only requires that surface depth can be evaluated at a screen sample point

But what about semi-transparent surfaces?

CMU 15-462/662 Compositing

CMU 15-462/662 Representing opacity as alpha

Alpha describes the opacity of an object - Fully opaque surface: ! = 1 - 50% transparent surface: ! = 0.5 - Fully transparent surface: ! = 0

Red triangle with decreasing opacity

! = 1 ! = 0.75 ! = 0.5 ! = 0.25 ! =0

CMU 15-462/662 Alpha: additional channel of image (RGBA)

! of foreground object

CMU 15-462/662 Over operator:

Composite image B with opacity !B over image A with opacity !A

A A over B != B over A B B “Over” is not commutative A B over A A over B

Koala over NYC CMU 15-462/662 Fringing Poor treatment of color/alpha can yield dark “fringing”:

foreground color foreground alpha background color

fringing no fringing CMU 15-462/662 No fringing

CMU 15-462/662 Fringing (…why does this happen?)

CMU 15-462/662 Lecture 5 Math Lecture 5 Math

d (x, y)=Ax + By + C wd (x, y)=Ax + By + C w T p0x p0y ,d0 T p0x p0y T ,d0 ⇥p1x p1y⇤ ,d1 T ⇥p1x p1y⇤T ,d1 ⇥p2x p2y⇤ ,d2 LectureT 5 Math ⇥p2x p2y⇤⇤ ,d2 C =⇥↵ B + (1⇤ ↵ )↵ A B B A C = ↵BB +d (1 ↵B)↵AA Over operator:(x, y)= non-premultipliedAx + By + C alpha ↵ = ↵ + (1 ↵ )↵ C B w B A Composite image B with opacity !B over image A with opacity !A ↵C = ↵B + (1 ↵B)↵A A frst attempt: T T A = Ar Ag pA0xb p0y ,d0 T A = ⇥A A A ⇤ r g⇥ b ⇤T A p1x T p1y ,d1 B B = ⇥Br Bg Bb ⇤ T T B = ⇥B B⇥p2Bx⇤ p2y⇤ ,d2 B over A r g b Appearance of semi- transparent A Composited⇥ color: ⇥ ⇤ ⇤ C = ↵ B + (1 ↵ )↵ A B B A

Appearance of What B lets through semi-transparent B ↵ = ↵ + (1 ↵ )↵ C B B A

CMU 15-462/662 Lecture 5 Math Lecture 5 Math

d (x, y)=Ax + By + C wd (x, y)=LectureAx + By + 5C Math w T p0x p0y ,d0 T p d p T ,d ⇥p0x (px,0y y⇤)=,dAx0 + By + C 1xw 1y 1 ⇥ ⇤T pLecture1x p1y T ,d 5 Math1 ⇥p2x p2y⇤ ,d2 ⇥ ⇤T T ⇥p2x pp2y0⇤x ,dp0y2 ,d0 T C =⇥d↵BB +⇥ (1⇤ ↵B)↵⇤AA (x, y)=p1xAx p+1yBy,d+ C1 COver=w↵ B +operator: (1 ↵ )↵ A premultiplied alpha B ⇥ B ⇤TA ⇥p2x p2y⇤ ,d2 Composite↵C = ↵B + image (1 ↵ TBB with)↵A opacity B over image A with opacity A p0x⇥ p0y ,d⇤ 0 ! ! Not premultiplied: ↵C Lecture= ↵B + (1 5↵ MathB)↵A C⇥ = ↵ B⇤+T (1 ↵ )↵ A Non-premultipliedCp=1x ↵BpB1y + alpha: (1,dT 1↵B)↵AA A = Ar Ag Ab ↵ = ↵ +T (1 ↵ )↵ ⇥p↵C =p↵B⇤+,d (1T ↵B)↵A dA = ⇥A2rx Ag2y Ab⇤ 2 (x, y)=Ax + By + C T A w A = Ar ATg Ab B B =⇥⇥Br Bg ⇤Bb ⇤ ⇥ T ⇤T BC == ⇥↵BBBB=B+⇥B (1rB ⇤B↵Bg )↵BAbA⇤ two multiplies, one add r g T b B over A p0x p0y ,d0 (referring to vector ops on colors) Premultiplied: ⇥ ⇥ ⇤ ⇤ C0 = BT+ (1 ↵B)A Premultiplied⇥p1x 0 p1 alpha:y⇤ ,d1 B ↵C = ↵B + (1 ↵B)↵A T A0 = ↵AAr ↵TAAg ↵AAb ↵A ⇥p2x p2y⇤ ,d2 T B = ⇥↵ B ↵ B ↵ B ↵ ⇤ T B0 =⇥ ↵BBr ⇤↵BBg ↵BBb ↵B ⇥ ⇤ one multiply, one add C = ↵⇥ B + (1 ↵ )↵ A ⇤ B B A Composite alpha: Notice premultiplied alpha composites alpha just like how it composites rgb. ↵C = ↵B + (1 ↵B)↵A Non-premultiplied alpha composites alpha differently than rgb.

CMU 15-462/662 A problem with non-premultiplied alpha

! Suppose we upsample an image w/ an alpha mask, then composite it onto a background ! How should we compute the interpolated color/alpha values? ! If we interpolate color and alpha separately, then blend using the non-premultiplied “over” operator, here’s what happens:

original original color alpha

upsampled upsampled color alpha

Notice black “fringe” that occurs because we’re blending, e.g., 50% blue pixels using 50% alpha, rather than, say, 100% blue pixels with 50% alpha.

composited onto yellow background CMU 15-462/662 Eliminating fringe w/ premultiplied “over”

! If we instead use the premultiplied “over” operation, we get the correct alpha:

(1-alpha) background

+ =

upsampled color (1-alpha)*background composite image w/ no fringe

CMU 15-462/662 Eliminating fringe w/ premultiplied “over”

! If we instead use the premultiplied “over” operation, we get the correct alpha:

composite image WITH fringe

CMU 15-462/662 Similar problem with non-premultiplied alpha Consider pre-fltering (downsampling) a texture with an alpha matte Desired fltered result

!

input color input ! fltered color fltered ! fltered result Downsampling non-premultiplied alpha composited over white image results in 50% opaque brown)

0.25 * ((0, 1, 0, 1) + (0, 1, 0, 1) + Result of fltering (0, 0, 0, 0) + (0, 0, 0, 0)) = (0, 0.5, 0, 0.5) premultiplied image

CMU 15-462/662 LectureLecture 5 Math 5 Math Lecture 5 Math

d d Lecture(x, y)=Ax 5+ MathBy + C (x,wd y)=Ax + By + C w (x, y)=Ax + By + C w T d p0x p0y ,d0 (x,Lecture y)=TAx +T 5By Math+ C pw0x pp0x0y p0,dy T ,d0 0 ⇥p1x p1y⇤ ,d1 T T ⇥p ⇥pp ⇤p ,d⇤T ,d 1x ⇥pLecture1x1y p1y⇤T ,d 51 Math1 dp02xx p02yy ,d02 ⇥(x, y)=T ⇤AxT + By + C ⇥p2x ⇥pp2x2y⇤p2,dy⇤T ,d2 2 w⇥p1x p1y⇤ ,d1

⇥C =⇥d↵BB⇤+ (1⇤T ↵B)↵AA ⇥p2(xx, yp)=2y⇤ Ax,d+2By + C Not premultiplied: Morew problems:T applying “over” repeatedly C = ↵BpB0x+ (1p0y ↵B,d)↵AA0 C = ↵BB⇥ + (1 ⇤↵B)↵AA Not premultiplied: T ↵C =⇥p↵B +p (1 ⇤↵TB,d)↵A ↵C C=Composite=↵B↵B+B1 (1x+ image (1↵1yB↵) BC↵ )withA↵AA1 opacity !C over B with opacity !B over image A with opacity !A p0x p0y ,d0 ↵ = ↵ + (1 ↵ )↵ C = ↵ B+ (1 ↵TTB)↵ A A =↵C A⇥p⇥ B2Ax p2Ay⇤⇤TB,dA 2 Non-premultipliedr p1xg p1yb alpha,dT 1 is not closed under composition: A = Ar Ag Ab T A = Ar Ag ATb ⇥ ⇥ ⇤⇤T T B = Br⇥pB2xg pB2yb⇤ ,d2 BA A = ⇥Ar Ag Ab⇤T B = ⇥B B B ⇤ Premultiplied: C⇥= ↵BrB +g (1⇤ b ↵TB)↵AA B =⇥⇥Br Bg ⇤Bb ⇤ Premultiplied: ⇥ ⇤ C0 = B + (1 ↵B)A T C CBC0 == B⇥↵BB+B (1B+ (1↵B)⇤A↵B)↵AA r g Bb T A0 = ↵ A ↵ A ↵ A ↵ A r↵C =A ↵gB +A (1 b ↵BA)↵AT A0 = ↵AAr ⇥ ↵AAg ↵AA⇤b ↵A T C over B over A B0 = ⇥↵ B ↵ B ↵ B ↵ ⇤ T B B= ⇥↵r ↵BCB=↵g↵BB +B (1↵ bB↵BB↵)↵⇤A 0 ConsiderB r resultB g of Bcompositing b B 50% red over 50% red: T ⇥ T ⇤ C⇥=C0=.75000.7500 Wait…⇤ this result is the premultiplied color! “Over” for non-premultiplied alpha takes non-premultiplied colors to premultiplied ↵⇥ =0⇥ .75 ⇤ ⇤ C colors (“over” operation is not closed) Cannot compose “over” operations on non-premultiplied values: over(C, over(B, A))

Q: What would be the correct UN-premultiplied RGBA for 50% red on top of 50% red?

CMU 15-462/662 Summary: advantages of premultiplied alpha

▪ Simple: compositing operation treats all channels (RGB and A) the same ▪ More efficient than non-premultiplied representation: “over” requires fewer math ops ▪ Closed under composition ▪ Better representation for fltering (upsampling/ downsampling) textures with alpha channel

CMU 15-462/662 Strategy for drawing semi-transparent primitives Assuming all primitives are semi-transparent, and RGBA values are encoded with premultiplied alpha, here’s one strategy for creating a correctly rasterized image:

over(c1, c2) { return c1.rgba + (1-c1.a) * c2.rgba; }

update_color_buffer( x, y, sample_color, sample_depth ) { if (pass_depth_test(sample_depth, zbuffer[x][y]) { // (how) should we update depth buffer here?? color[x][y] = over(sample_color, color[x][y]); } }

Q: What is the assumption made by this implementation? Triangles must be rendered in back to front order!

CMU 15-462/662 Putting it all together Now what if we have a mixture of opaque and transparent triangles?

Step 1: render opaque primitives (in any order) using depth-buffered occlusion If pass depth test, triangle overwrites value in color buffer at sample

Step 2: disable depth buffer update, render semi-transparent surfaces in back-to-front order. If pass depth test, triangle is composited OVER contents of color buffer at sample

CMU 15-462/662 End-to-end rasterization pipeline (“real-time graphics pipeline”)

CMU 15-462/662 Lecture 5 Math Lecture 5 Math d (x, y)=Ax + By + C w d (x, y)=Ax + By + C w T p0x p0y ,d0 T T p0x ⇥pp10xy p,d1y⇤ 0,d1 T T ⇥p1x ⇥pp21xy⇤ p,d2y⇤ 1,d2 ⇥ ⇥ ⇤T ⇤ Not premultiplied: p2x p2y ,d2 ⇥C = ↵BB⇤+ (1 ↵B)↵AA Not premultiplied: 1 C C= =↵BB +(↵ (1BB +↵B (1)↵A↵AB)↵AA) ↵C 1 C = (↵↵CB=B +↵B (1+ (1↵B)↵↵BA)A↵)A ↵C T ↵ = ↵A =+ (1Ar ↵Ag )↵Ab C B B A ⇥ T ⇤T Goal: turnA these= BA=r ABinputsgr BAgb Bintob an image! Premultiplied: ⇥ ⇥ ⇤T ⇤ Inputs: B = Br Bg Bb C0 = B + (1 ↵B)A ⇥ ⇤ Premultiplied: list_of_positions = { list_of_texcoords = { T A0C=0 =↵BAA+r (1↵AA↵gB)A↵AAb ↵A v0x, v0y, v0z, v0u, v0v, v1x, v1y, v1x, ⇥ v1u, v1v, T ⇤ T v2x, Av2y,0 = v2z,B0↵=AA↵r BB↵Ar A g↵v2u,BB↵ v2v,Ag Ab↵B↵BAb ↵B v3x, v3y, v3x, v3u, v3v, T T ⇥ v4u, v4v, ⇤ v4x, Bv4y,0 = v4z,⇥↵BBr C↵=BB0g.7500↵BBb ↵B⇤ Texture map v5x, v5y, v5x }; v5u, v5v }; T ⇥ C = 0.7500↵⇥C =0.75 ⇤ ⇤ Object-to-camera-space transform: T = P ↵⇥C =0.75 ⇤

Perspective projection transformT = P

Size of output image (W, H)

At this point we should have all the tools we need, but let’s review…

CMU 15-462/662 Step 1: Transform triangle vertices into camera space

y

z

x

CMU 15-462/662 T T x = xx/ xz xy/ xz T x2D =2Dxx/ xz xy/ xz Step 2: x2D = xx/ xz xy/ xz ⇥ ⇥ ⇤ ⇤ T ⇥x = x / x x⇤ / x Apply perspective projection2D x z ytransform z to transform triangletan(✓tan(/2) vertices✓/2) tan(✓⇥/2) ⇤ T x2DT= xx/ xz xy/ xz x2D = xx/ xz xy/ xz T into normalized coordinatetan(✓/ 2)space T x2D = xx/ xz xy/ xz aspect⇥ aspecttan(✓tan(/2)✓/⇤2) ⇥ ⇤ x⇥2D = xx/ xz xy/ xz T aspect tan(✓/2) ⇥ x = x / x x / x ⇥ ⇥ ⇤ tan(✓/2) 2D x z y z tan(✓/2) ⇥ ⇤ aspect tan(✓/2) y ⇥ ⇤ tan(✓/2) x1 x2 x3 x4 x5 x6 x7 x8 ⇥ T T x1 x2 x3 x4 x5 x6 xtan(7✓x/2)8 x1 x2 x3 x4 xx5 2D x=6 xx2Dx/7= xxz 8x/xyx/z xxz y/ xz tan(✓/2) aspect tan(✓/2) aspect tan(✓/2) ⇥ Lecture 3 Math ⇥ ⇥ ⇤ ⇤ T ⇥ y x1 x2 x3 x4 x5 x6x2Dx=7 xxx/8 xz xy/ xz tan(✓tan(/2)✓/2) aspect tan(✓/2) aspect tan(✓/2) tan(✓/2) tan(✓/2)tan(⇥✓/2) T aspect ⇥ x2D = ⇥xx/ xz xy/ xz ⇤ aspect aspect tan(✓/2) aspect x1 x2 x3 x4 x5 xT6 x7 x8⇥ T x1 x2 x3 x4 x5 x2Dx6= xx7/ xx8xz 2Dx=y/ xxxz/ xz xy/(1, x1,z 1) tan(✓/2) ⇥ tan(✓/2) ⇤ z x1 x2 x3 x4 x5 x6 x7 x8 aspect ⇥ x1 ⇥x2 ⇤x3 x4 x5 ⇤ x6 x7 x8 aspect aspecttan( ✓/2)tan(✓/2) x0 x1 x2 x3 y tan(✓/2) (-1,-1,-1)tan(✓/x2)1 x2 x3 x4 x5 x6 x7 x8 ⇥ ⇥ tan(✓/2) tan(✓/aspect2) tan(✓/2) tan(aspect✓/2) tan(✓/2) aspect z x + y ✓ ⇥ tan(✓/2) x1 x2x1 x3x2 x4x3 x5x4aspectx6x5 x7x6 x8x7 x8 tan(✓/2) aspect tan(✓/2) aspect ⇥ aspect tan(✓aspect/2) tan(✓/2)aspect x Pinhole x1 x2 x3 x4 x5 x6 x7 x8 ⇥ x ⇥ Camera tan(✓/2)tan(✓/2) x1 x2 x3 x4 x5 x6 x7 x8 f(x) znearaspect aspect (0,0) x1 x2 x3 xx4 xx5 xx6 xx7 xx8 x x x tan(✓/2) 1 2 3 4 5 6 7 8 Camera-space positions: 3D tan(aspect✓/2) Normalized space positions aspect tan(✓/2) tan(✓/2) f(x + y)=f(x)+f(y) aspect aspect

f(ax)=af(x)

CMU 15-462/662 f(x)=af(x)

Scale: Sa(x)=ax

S (x) S (x ) S (x ) S (x S (ax) aS (x) S (x) S (y) S (x + y) 2 2 1 2 2 2 3 2 2 2 2 2

S (x)=2x 2 2 2 2

aS2(x)=2ax 2

S2(ax)=2ax 2 2 S (ax)=aS (x) 2 2 2 2 2

S2(x + y) = 2(x + y) 2 2 S2(x)+S2(y)=2x +2y 2 S2(x + y)=S2(x)+S2(y) 2 2 2 Rotations:

R (x) R (x ) R (x ) R (x ) R (x ) R (ax) aR (x) R (y) R (x + y) ✓ ✓ 0 ✓ 1 ✓ 2 ✓ 3 ✓ ✓ ✓ ✓ Translation: T (x ) T (x ) T (x ) T (x ) a,b 0 a,b 1 a,b 2 a,b 3 Step 3: clippingT T x = x / x x / x T x2D =2Dxx/ xxz xzy/ xyz z T x =x2Dx=/ xxx/ xz / xxy/ xz 2D x z y z ▪ Discard ⇥triangles⇥ that lie⇤ complete⇤ outside the unit cube (culling) ⇥ ⇥ ⇤ ⇤ tan(✓/2) - They aretan( o✓ff/ 2)screen, don’t bother processing them further tan(✓/2) T tan(✓/2) x2D = xx/ xz xy/ xz T ▪ Clip triangles that extend beyond the unit cube to thex cube2D = xx/ xz xy/ xz T aspect⇥ aspecttan(x ✓tan(/=2)✓x/⇤2)/ x x / x - (possibly ⇥generating⇥2D x newz triangles)y z T aspect⇥ aspecttan(✓tan(/2)✓/⇤2) T x = x / x x / x x⇥2D = xx/ xz xy/ xz T tan(✓/2) 2D x z y z ⇥ x = x / x x / x y ⇥ ⇤ tan(✓/2) 2D x z y z ⇥ ⇤ y ⇥ ⇤ x1 x2 x3 x4 x5 x6 x7 x8 ⇥ ⇤ x1 x2 x3 x4 x5 x6 xtan(7✓x/2)8 x x x x x x x x tan(✓/2) x1 x12 x23 x34 x45 x56 xtan(67✓x/72)8 8 aspect tan(✓/2) tan(✓/2) ⇥ aspect tan(✓/2) tan(✓tan(/2)✓/2) ⇥ aspect tan(✓/2) tan(✓tan(/2)✓/2) aspectaspect aspect⇥ tan(✓/2) aspect tan(✓/2) x1 x2 x3 x4 x5 xT6 x7 x8⇥ T aspectaspect aspect⇥ tan(✓/2) x2D= xx/ xxz 2Dx=y/ xxxz/ xz xy/(1, 1,x 1)z x1 x2 x3 x4 x5 xT6 x7 x8⇥ T z x2D= xx/ xxz 2Dx=y/ xxxz/ xz xy/(1, 1,x 1)z x x x x x x x x z ⇥ 1 ⇥ 2 ⇤ 3 4 5 ⇤ 6 7 8 x x x x x x x x (-1,-1,-1)tan(✓/x2)1 x2 x3 x4 x5 x6 x7 x8 ⇥ 1 ⇥ 2 ⇤ 3 4 5 ⇤ 6 7 8 (-1,-1,-1)tan(✓/x2)1 x2 x3 x4 x5 x6 x7 x8 tan(✓/2) tan(✓/2) aspect tan(✓/2) tan(✓/2) tan(✓/2) aspect tan(✓/2) tan(✓/2) aspect tan(✓/2) aspect tan(✓aspect/2) tan(✓/2)aspect aspect ⇥ x ⇥ aspect tan(✓aspect/2) tan(✓/2)aspect ⇥ x ⇥ x1 x2 x3 xx1 4 xx2 5 xx3 6 xx4 7 xx5 8 x6 x7 x8 x1 x2 x3 xx4 xx5 xx6 xx7 xx8 x x x 1 2 3 4 5 6 7 8

tan(✓/2) tan(✓/2) Triangles after clipping Triangles before clipping tan(✓/2) tan(✓/2) aspect aspect aspect aspect CMU 15-462/662

2 2 2 2

2 2 2 2 2 2

2 2 2 2 Step 4: transform to screen coordinates

Perform homogeneous divide, transform vertex xy positions from normalized coordinates into screen coordinates (based on screen w,h)

(w, h)

(0, 0)

CMU 15-462/662 Lecture 5 Math LectureLecture 5 Math 5 Math Lecture 5 Math d (x, y)=Ax + By + C LectureLecture 5 Math 5 Math d w d (x, y)=Ax + By + C (x, yw)=Ax + By + C Lecture 5 Mathd w T (x, y)=Ax + By + C p0x p0y ,d0 d d w (x, y)=(x,Ax y)=+ AxBy++ByC + C T T p0x Tp0y ⇥ ,d0 ⇤ w w p0x p0y ,dp0 1x p1y ,d1 d T T p p ,d⇥ T ⇤ T (x, y)=Ax + By + C0x 0y ⇥ 0 p1x⇤ p1y ⇥p,dp1 ⇤ ,d w T T p1x p1y ,d1 2x 2y 2 p0x pp00yx p,d0y 0 ,d0T T ⇥p p ⇤ ,d⇥ T ⇤ 1x 1y ⇥p 1 pp2x⇤ p,d2y ⇥ ,d2 ⇤ Not premultiplied:T T 2x 2y 2 ⇥p ⇥Tp ⇤ ,d⇤ T p0x p10xy p,d11yx ⇥0p1y 1 ,d⇤1 p2x p2y ⇥,d2⇥ ⇤ C =⇤ ↵BB +(1 ↵B)↵AA Not premultiplied: T Not premultiplied:⇥ ⇥T ⇤ ⇤T ⇥p pp2x ⇤ pp,d22yx p,d2y 2 ,d2 1 1x 1y ⇥1 C⇤= ↵C B= +(1↵BCB =+(1↵ )↵(↵↵ABB)↵+(1AA ↵ )↵ A) Not premultiplied: B B AB B A ⇥ ⇥T ⇤ ⇤ 1 ↵C ⇥p p ⇤ ,dC = ↵BB +(1 1 ↵B)↵AA Not premultiplied:Not premultiplied: 2x 2y 2 C = C =(↵ B(+(1↵BB +(1↵ )↵↵AB)↵AA) ↵BC ↵C B=↵AB +(1 ↵B)↵A C = ↵CBB= +(1↵BB1+(1↵B)↵A↵AB)↵↵CAA ⇥ ⇤C = (↵BB +(1 ↵B)↵AA) T Not premultiplied: ↵C = ↵B +(1A = ↵AB)↵AA A 1 1 ↵C ↵C = ↵B +(1 ↵B)↵A r g b C C= =↵ BC +(1=(↵BB(+(1↵↵BB)↵+(1↵AB)↵A↵AB))↵AA) T B↵ B A T T C ↵C ↵C = ↵B +(1 ↵BA)↵=A Ar BAg= ⇥ABb B B ⇤ 1 A= Ar Ag Ab r g b C = (↵↵C B=↵+(1↵B =+(1↵↵ +(1)↵↵B)A↵)A↵ )↵ T T B C BAB= AAr BAg AA ⇥ T ⇤ ↵C Premultiplied: ⇥bB = Br Bg⇤ ⇥Bb ⇤ T B = Br Bg Bb A = A A A T ↵ = ↵ +(1Ar=↵Ag)r↵ A⇥bg Ab ⇤T C0 = B +(1 ↵B)A Premultiplied:Premultiplied:C B BB =A Br Bg B⇥b ⇥ ⇤ ⇤ T T ⇥ T ⇤ T C0 = B +(1 ↵ )A B = BBr= ⇥BBg BBb B ⇤C0 = B +(1A0 = ↵↵A)AAr B↵AAg ↵AAb ↵A Premultiplied: A = Ar Ag Ab r ⇥ g b ⇤ B T T C0T= B +(1A0 =↵B↵)AAAr ↵A⇥Ag ↵AAb T ↵A ⇤ Premultiplied:Premultiplied: ⇥ ⇥ A⇤0 = ↵⇤AAr ↵ABA0 g= ↵↵ABABr ↵↵ABBg ↵BBb ↵B B = ⇥Br Bg Bb⇤ b C0 = BC +(1= B +(1↵B)A↵ )A T T A 0= ↵ A ↵ AB ↵⇥ A ↵ T ⇤ T 0 A r A B⇥ g0 = A↵BbBr A↵B⇥ BCg =↵B0B.75b⇤ ↵ 0B 0 ⇤ Premultiplied: ⇥ ⇤ B0 = ↵BBT r ↵TBBg ↵BBb ↵B A0 = A↵0A=Ar↵A↵AArAg↵A↵AAgAb ↵A↵AAb ↵A T T C0 = B +(1B = ↵⇥↵B)AB ↵ B ↵⇥ B ↵ ⇤ T ⇤ Step 5: setup triangle0 B (triangler B ⇥ g preprocessing)C =B C0b.75= B0 0.75 0 0↵ 0⇥C =0⇤ .75 ⇤ ⇥ T ⇤ T T B0 = B↵0B=Br⇥↵B↵BBrBg↵B↵BBgBb↵B↵BBb ↵BT⇤ BeforeA0 = rasterizing↵AA triangle,r ↵ canAA computeg ↵⇥ aA bunchACb= ↵0A.75 0 0 ↵⇥C⇤=0.75 ⇤T = P ↵⇥C =0.75 ⇤ of data that will be used by all fragments, e.g., T T ⇥ ⇥ C =⇥ C0.75= 0 0.75 0 0⇤ T 0 ⇤ ⇤ B• triangle0 = edge↵B equationsBr ↵ BBg ↵BBb ↵↵⇥CB =0.75 ⇤ T = PT = P • triangle attribute equations ↵ =0.75 ⇥ ⇥C ↵⇥CT=0⇤.75T⇤=⇤P E (x, y)=E (x, y)=E (x, y) • etc. C = 0.75 0 0 01 12 20 T = PT = P E01(x, y)=E12(x, y)=E20(x,1 y) ↵⇥C =0.75 ⇤ E01(x, y)=EU12((x,x, y y)=)=VE(20x,(x, y)= y) (x, y)=Z(x, y) 1 w T =EP01(x, y)=E12U(x,(x, y y)=)=EV20((x,x, y1 y)=) (x, y)=Z(x, y) U(x, y)=V(x, y)= (x, yw)=Z(x, y) E01(x,E y)=(x,E y)=12(x,E y)=(x,E y)=20(x,E1 y) (x, y) w 01U(x, y)=12V(x, y)= 20(x, y)=Z(x, y) 1 1 w E U((x,x, y yU)=)=(x,EV y)=(x,(x, yV y)=)=(x, yE)=(x,(x, y)= y(x,) Z y()=x, yZ) (x, y) 01 12 w 20 w 1 U(x, y)=V(x, y)= (x, y)=Z(x, y) w

CMU 15-462/662 Step 6: sample coverage Evaluate attributes z, u, v at all covered samples

CMU 15-462/662 Step 6: compute triangle color at sample point e.g., sample texture map *

u(x,y), v(x,y) v

u

* So far, we’ve only described computing triangle’s color at a point by interpolating per-vertex colors, or by sampling a texture map. Later in the course, we’ll discuss more advanced algorithms for computing its color based on material properties and scene lighting conditions. CMU 15-462/662 Step 7: perform depth test (if enabled) Also update depth value at covered samples (if necessary)

PASS

PASS PASS FAIL PASS PASS FAIL PASS PASS PASS

FAIL FAIL PASS PASS PASS FAIL FAIL PASS PASS PASS

CMU 15-462/662 Step 8: update color buffer* (if depth test passed)

* Possibly using OVER operation for transparency; in general there are more compositing/raster operations that can be used to compute the new color value. CMU 15-462/662 OpenGL/Direct3D graphics pipeline * Structures rendering computation as a series of operations on vertices, primitives,

fragments, and screen samples 3 1 4 Input: vertices in 3D space 2 Operations on Vertex Processing vertices Vertex stream Vertices in positioned in 3D normalized Operations on Primitive Processing coordinate space primitives (triangles, lines, etc.) Primitive stream Fragment Generation Triangles projected to 2D screen (Rasterization)

Operations on Fragment stream fragments Fragments (one fragment per covered sample) Fragment Processing

Shaded fragment stream Shaded fragments Operations on Screen sample operations (depth and color) screen samples Output: image (pixels)

* Several stages of the modern OpenGL pipeline are omitted CMU 15-462/662 OpenGL/Direct3D graphics pipeline *

3 1 4 Input vertices in 3D space 2

Operations on Vertex Processing transform matrices vertices Vertex stream

Operations on Primitive Processing primitives (triangles, lines, etc.) Primitive stream textures Fragment Generation (Rasterization) Operations on Fragment stream Pipeline inputs: fragments Fragment Processing - Input vertex data (positions, colors, UVs)

Shaded fragment stream - Parameters needed to compute vertex coordinates in 3D space (e.g., transform matrices) Operations on Screen sample operations - Parameters needed to compute color of fragments (depth and color) screen samples (e.g., textures) - “Shader” programs that defne behavior of vertex and fragment stages

* several stages of the modern OpenGL pipeline are omitted CMU 15-462/662 Programmable Shaders Defne behavior of vertex processing and fragment processing stages Describe operation on a single vertex (or single fragment)

Example GLSL fragment shader program Shader function executes once uniform sampler2D myTexture; constant parameters per fragment. uniform vec3 lightDir; per-fragment attributes varying vec2 uv; (interpolated by rasterizer) Outputs color of surface at varying vec3 norm; sample point corresponding to fragment. void diffuseShader() (this shader performs a texture lookup to { grab color from texture obtain the surface’s material color at this point, vec3 kd; then performs a simple lighting computation) kd = texture2d(myTexture, uv); kd *= clamp(dot(-lightDir, norm), 0.0, 1.0); gl_FragColor = vec4(kd, 1.0); } output surface color at 100% opacity (1.0) modulate color according to light/normal direction

CMU 15-462/662 Goal: render very high complexity 3D scenes - 100’s of thousands to millions of triangles in a scene - Complex vertex and fragment shader computations - High resolution screen outputs (~10Mpixel + supersampling) - 30-120 fps

Unreal Engine Kite Demo (Epic Games 2015) CMU 15-462/662 Graphics pipeline implementation: GPUs Specialized processors for executing graphics pipeline computations

Discrete GPU card (NVIDIA GeForce Titan X)

Integrated GPU: part of modern Intel CPU die CMU 15-462/662 GPU: heterogeneous, multi-core processor still enormous amount of fxed- Modern GPUs offer ~13 TFLOPs of performance for function compute over here generic vertex/fragment programs (“compute”)

SIMD SIMD SIMD SIMD Texture Texture Exec Exec Exec Exec

Cache Cache Cache Cache Texture Texture

SIMD SIMD SIMD SIMD Tessellate Tessellate Exec Exec Exec Exec

Tessellate Tessellate Cache Cache Cache Cache GPU Clip/Cull Clip/Cull Rasterize Rasterize Memory SIMD SIMD SIMD SIMD Exec Exec Exec Exec Clip/Cull Clip/Cull Rasterize Rasterize Cache Cache Cache Cache

Zbuffer / Zbuffer / Zbuffer / Blend Blend Blend SIMD SIMD SIMD SIMD Exec Exec Exec Exec Zbuffer / Zbuffer / Zbuffer / Blend Blend Blend Cache Cache Cache Cache Scheduler / Work Distributor

This part (mostly) not used by CUDA/OpenCL; raw graphics horsepower still greater than compute! CMU 15-462/662 Modern Rasterization Pipeline ▪ Trend toward more generic (but still highly parallel!) computation: - more programmable stages - replace fxed function vertex, fragment processing - add geometry, tessellation shaders - generic “compute” shaders (whole other story…) - more fexible scheduling of stages

(DirectX 12 Pipeline) CMU 15-462/662 Ray Tracing in Graphics Pipeline ! New pipelines coming down the pipe…

https://devblogs.nvidia.com/introduction-nvidia-rtx-directx-ray-tracing/

CMU 15-462/662 GPU Ray Tracing Demo (“Refections”)

Great video on how this was done: https://youtu.be/JAKXySYfLWo CMU 15-462/662 Summary

▪ Occlusion resolved independently at each screen sample using the depth buffer ▪ Alpha compositing for semi-transparent surfaces - Premultiplied alpha forms simply repeated composition - “Over” compositing operations is not commutative: requires triangles to be processed in back-to-front (or front-to-back) order

▪ Graphics pipeline: - Structures rendering computation as a sequence of operations performed on vertices, primitives (e.g., triangles), fragments, and screen samples - Behavior of parts of the pipeline is application-defned using shader programs. - Pipeline operations implemented by highly, optimized parallel processors and fxed-function hardware (GPUs)

CMU 15-462/662 What else do we need to know to generate images like these? GEOMETRY How do we describe complex shapes (so far just triangles…)

RENDERING How does light interact w/ materials to produce color?

ANIMATION (“Moana”, Disney 2016) How do we describe the way things move?

CMU 15-462/662 Course roadmap

Introduction

Drawing a triangle (by sampling) Drawing Things Transforms and coordinate spaces Perspective projection and texture sampling

Today: putting it all together: end-to-end rasterization pipeline

Geometry Next time!

Materials and Lighting

CMU 15-462/662