Buffers, Compositing and Blending Week 8

CS 432/680 INTERACTIVE COMPUTER GRAPHICS Buffers, Compositing and Blending Week 8

David Breen Department of Computer Science Drexel University Based on material from Ed Angel, University of New Mexico Han-Wei Shen, Ohio State University David Blythe, Silicon Graphics & Tom McReynolds, Gigapixel Objectives

• Introduce additional OpenGL buffers • Learn to read and write buffers • Learn to use blending • Learn depth-of-field and reflection effects

Angel: Interactive Computer Graphics 3E 2 © Addison-Wesley 2002 Pixel and Geometry Pipelines

• Parallel pipelines (Geometry & Pixel) come together in the Rasterization stage

Angel: Interactive Computer Graphics 3E 3 © Addison-Wesley 2002 Buffer

Define a buffer by its spatial resolution (n x m) and its depth k, the number of bits/pixel

pixel

Angel: Interactive Computer Graphics 3E 4 © Addison-Wesley 2002 OpenGL Frame Buffer

Angel: Interactive Computer Graphics 3E 5 © Addison-Wesley 2002 OpenGL Buffers

• Color buffers can be displayed – Front – Back – Auxiliary – Overlay • Depth • Accumulation – High resolution buffer • Stencil – Holds masks

Angel: Interactive Computer Graphics 3E 6 © Addison-Wesley 2002 Writing in Buffers • Conceptually, we can consider all of memory as a large two-dimensional array of pixels • We read and write rectangular block of pixels – Bit block transfer (bitblt) operations • The frame buffer is part of this memory

memory source frame buffer (destination) writing into frame buffer Angel: Interactive Computer Graphics 3E 7 © Addison-Wesley 2002 The Pixel Pipeline

• OpenGL has a separate pipeline for pixels – Writing pixels involves • Moving pixels from processor memory to the frame buffer • Unpacking (Format conversions, swapping & alignment) • Transfer ops, Map Lookups, Tests – Reading pixels • Packing (Format conversion, swapping & alignment)

Angel: Interactive Computer Graphics 3E 8 © Addison-Wesley 2002 Pixel Storage Modes • Deal with byte-ordering conventions • Memory alignment glPixelStore[if](GLenum pname, TYPE param); pname: GL_UNPACK_SWAP_BYTES, GL_PACK_SWAP_BYTES, GL_UNPACK_LSB_FIRST, GL_PACK_LSB_FIRST, GL_UNPACK_ROW_LENGTH, GL_PACK_ROW_LENGTH, GL_UNPACK_SKIP_ROWS, GL_PACK_SKIP_ROWS, GL_UNPACK_ALIGNMENT, GL_PACK_ALIGNMENT, GL_UNPACK_ALIGNMENT Angel: Interactive Computer Graphics 3E 9 © Addison-Wesley 2002 Pixel Transfer Operations

• Color Scale & Bias • Color Index Shift & Offset • Convolution • Matrix multiply glPixelTransferf(GL_RED_SCALE, s); glPixelTransferf(GL_BLUE_BIAS, b);

Angel: Interactive Computer Graphics 3E 10 © Addison-Wesley 2002 Pixel Mapping Operations

• Color components and color & stencil indices can be modified via a table look-up glPixelMap[ui us f]v(Glenum map, Glint size, TYPE *array);

map - GL_PIXEL_MAP_*_TO_* I, S, R, G, B, A

Angel: Interactive Computer Graphics 3E 11 © Addison-Wesley 2002 Pixel Tests

• Scissor glScissor() – Only draw in a rectangular portion of screen • Alpha glAlphaFunc() – Draw based on alpha value • Stencil glStencilFunc(), glStencilOp() – Draw based on stencil value • Depth glDepthFunc(), – Draw based on depth value

Angel: Interactive Computer Graphics 3E 12 © Addison-Wesley 2002 Writing Model

Read destination pixel before writing source

glLogicOp(GLenum op)

Angel: Interactive Computer Graphics 3E 13 © Addison-Wesley 2002 Logical Pixel Operations

Angel: Interactive Computer Graphics 3E 14 © Addison-Wesley 2002 Writing Modes • Source and destination bits are combined bitwise • 16 possible functions (one per column in table) AND replace XOR OR NOT(s)

Angel: Interactive Computer Graphics 3E 15 © Addison-Wesley 2002 XOR mode

• Applying twice returns bit back to original value • XOR is especially useful for swapping blocks of memory such as menus that are stored off screen

If S represents screen and M represents a menu the sequence

S ← S0 M ← S ⊕ M S ← S ⊕ M swaps the S and M

Angel: Interactive Computer Graphics 3E 16 © Addison-Wesley 2002 Raster Position • OpenGL maintains a raster position as part of the state • Set by glRasterPos*() – glRasterPos3f(x, y, z); • The raster position is a geometric entity (like a vertex) – Passes through geometric pipeline – Eventually yields a 2D position in screen coordinates – This position in the frame buffer is where the next raster primitive is drawn

Angel: Interactive Computer Graphics 3E 17 © Addison-Wesley 2002 Window Position

• In OpenGL 1.4 glWindowPos*() was defined • Allows you to specify current raster position in window coordinates

Angel: Interactive Computer Graphics 3E 18 © Addison-Wesley 2002 Buffer Selection • OpenGL can draw into or read from any of the color buffers (front, back, auxiliary) • Default to the back buffer • Change with glDrawBuffer and glReadBuffer • Note that format of the pixels in the frame buffer is different from that of processor memory and these two types of memory reside in different places – Need packing and unpacking – Drawing and reading can be slow

Angel: Interactive Computer Graphics 3E 19 © Addison-Wesley 2002 Bitmaps

• OpenGL treats 1-bit pixels (bitmaps) differently than multi-bit pixels (pixelmaps) • Bitmaps are masks which determine if the corresponding pixel in the frame buffer is drawn with the present raster color – 0 ⇒ color unchanged – 1 ⇒ color changed based on writing mode • Bitmaps are useful for raster text – GLUT_BIT_MAP_8_BY_13

Angel: Interactive Computer Graphics 3E 20 © Addison-Wesley 2002 Raster Color • Same as drawing color set by glColor*() • Fixed by last call to glRasterPos*() glColor3f(1.0, 0.0, 0.0); glRasterPos3f(x, y, z); glColor3f(0.0, 0.0, 1.0); glBitmap(……. glBegin(GL_LINES); glVertex3f(…..) • Geometry drawn in blue • Ones in bitmap use a drawing color of red

Angel: Interactive Computer Graphics 3E 21 © Addison-Wesley 2002 Drawing Bitmaps glBitmap(width, height, x0, y0, xi, yi, bitmap) offset from raster position increments in raster position after bitmap drawn first raster position

second raster position Angel: Interactive Computer Graphics 3E 22 © Addison-Wesley 2002 Example: Checker Board

GLubyte wb[2] = {0 x 00, 0 x ff}; GLubyte check[4096]; int i, j; for(i=0; i<64; i++) for (j=0; j<64, j++) check[i*8+j] = wb[(i/8+j/8)%2]; glBitmap( 64, 64, 0.0, 0.0, 0.0, 0.0, check);

Angel: Interactive Computer Graphics 3E 23 © Addison-Wesley 2002 Pixel Maps

• OpenGL works with rectangular arrays of pixels called pixel maps or images • Pixels are in one byte chunks – Luminance (gray scale) images 1 byte/pixel – RGB 3 bytes/pixel • Three functions – Draw pixels: processor memory to frame buffer – Read pixels: frame buffer to processor memory – Copy pixels: frame buffer to frame buffer

Angel: Interactive Computer Graphics 3E 24 © Addison-Wesley 2002 OpenGL Pixel Functions

glReadPixels(x,y,width,height,format,type,myimage) start pixel in frame buffer size type of pixels type of image pointer to processor memory GLubyte myimage[512][512][3]; glReadPixels(0,0, 512, 512, GL_RGB, GL_UNSIGNED_BYTE, myimage);

glDrawPixels(width,height,format,type,myimage) starts at current raster position

Angel: Interactive Computer Graphics 3E 25 © Addison-Wesley 2002 OpenGL Pixel Functions

glDrawPixels(width,height,format,type,myimage) starts at current raster position

Angel: Interactive Computer Graphics 3E 26 © Addison-Wesley 2002 OpenGL Pixel Functions

glReadPixels(x,y,width,height,format,type,myimage) start pixel in frame buffer size type of pixels type of image pointer to processor memory GLubyte myimage[786432]; glReadPixels(0,0, 512, 512, GL_RGB, GL_UNSIGNED_BYTE, myimage);

glDrawPixels(width,height,format,type,myimage) starts at current raster position

Angel: Interactive Computer Graphics 3E 27 © Addison-Wesley 2002 Formats & Types

• GL_RGB • GL_UNSIGNED_BYTE • GL_RGBA • GL_BYTE • GL_RED • GL_BITMAP • GL_GREEN • GL_UNSIGNED_SHORT • GL_BLUE • GL_SHORT • GL_ALPHA • GL_UNSIGNED_INT • GL_DEPTH_COMPONENT • GL_INT • GL_LUMINANCE • GL_FLOAT • GL_LUMINANCE_ALPHA • GL_UNSIGNED_BYTE_3_3_2 • GL_COLOR_INDEX • GL_UNSIGNED_INT_8_8_8_8 • GL_STENCIL_INDEX • etc.

Angel: Interactive Computer Graphics 3E 28 © Addison-Wesley 2002 Image Formats • We often work with images in a standard format (JPEG, TIFF, GIF) • How do we read/write such images with OpenGL? • No support in OpenGL – OpenGL knows nothing of image formats – Some code available on Web – Can write readers/writers for some simple formats in OpenGL Angel: Interactive Computer Graphics 3E 29 © Addison-Wesley 2002 Displaying a PPM Image

• PPM is a very simple format • Each image file consists of a header followed by all the pixel data • Header P3 # comment 1 # comment 2 . #comment n rows columns maxvalue pixels Angel: Interactive Computer Graphics 3E 30 © Addison-Wesley 2002 Reading the Header FILE *fd; int k, nm; char c; int i; char b[100]; float s; check for “P3” int red, green, blue; in first line printf("enter file name\n"); scanf("%s", b); fd = fopen(b, "r"); fscanf(fd,"%[^\n] ",b); if(b[0]!='P'|| b[1] != '3'){ printf("%s is not a PPM file!\n", b); exit(0); } printf("%s is a PPM file\n",b); Angel: Interactive Computer Graphics 3E 31 © Addison-Wesley 2002 Reading the Header (cont)

fscanf(fd, "%c",&c); while(c == '#') { fscanf(fd, "%[^\n] ", b); printf("%s\n",b); fscanf(fd, "%c",&c); } ungetc(c,fd);

skip over comments by looking for # in first column

Angel: Interactive Computer Graphics 3E 32 © Addison-Wesley 2002 Reading the Data fscanf(fd, "%d %d %d", &n, &m, &k); printf("%d rows %d columns max value= %d\n",n,m,k); nm = n*m; image=malloc(3*sizeof(GLuint)*nm); s=255./k; scale factor for(i=0;i

Angel: Interactive Computer Graphics 3E 33 © Addison-Wesley 2002 Scaling the Image Data

We can scale the image in the pipeline glPixelTransferf(GL_RED_SCALE, s); glPixelTransferf(GL_GREEN_SCALE, s); glPixelTransferf(GL_BLUE_SCALE, s);

We may have to swap bytes when we go from processor memory to the frame buffer depending on the processor. If so we need can use glPixelStorei(GL_UNPACK_SWAP_BYTES,GL_TRUE); Angel: Interactive Computer Graphics 3E 34 © Addison-Wesley 2002 The display callback void display() { glClear(GL_COLOR_BUFFER_BIT); glRasterPos2i(0,0); glDrawPixels(n,m,GL_RGB, GL_UNSIGNED_INT, image); glFlush(); }

Angel: Interactive Computer Graphics 3E 35 © Addison-Wesley 2002 Compositing and Blending Objectives

• Learn to use the A component in RGBA color for – Blending for translucent surfaces – Compositing images – Antialiasing

Angel: Interactive Computer Graphics 3E 37 © Addison-Wesley 2002 Opacity and Transparency • Opaque surfaces permit no light to pass through • Transparent surfaces permit all light to pass • Translucent surfaces pass some light translucency = 1 – opacity (α)

opaque surface α =1

Angel: Interactive Computer Graphics 3E 38 © Addison-Wesley 2002 Physical Models • Dealing with translucency in a physically correct manner is difficult due to – the complexity of the internal interactions of light and matter – Using a pipeline renderer – Revert to writing model

Angel: Interactive Computer Graphics 3E 39 © Addison-Wesley 2002 Writing Model

• Use A component of RGBA (or RGBα) color to store opacity • During rendering we can expand our writing model to use RGBA values blend source blending factor source destination component component

destination blending factor Color Buffer

Angel: Interactive Computer Graphics 3E 40 © Addison-Wesley 2002 Blending Equation

• We can define source and destination blending factors for each component

b = [br, bg, bb, bα] c = [cr, cg, cb, cα] source and destination colors

s = [sr, sg, sb, sα] d = [dr, dg, db, dα]

Blend as

pixel = [br sr+ cr dr, bg sg+ cg dg , bb sb+ cb db , bα sα+ cα dα ]

Angel: Interactive Computer Graphics 3E 41 © Addison-Wesley 2002 OpenGL Blending and Compositing

• Must enable blending and pick source and destination factors glEnable(GL_BLEND) glBlendFunc(source_factor, destination_factor) • Only certain factors supported – GL_ZERO, GL_ONE – GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA – GL_DST_ALPHA, GL_ONE_MINUS_DST_ALPHA – See Redbook for complete list

Angel: Interactive Computer Graphics 3E 42 © Addison-Wesley 2002 Example

• Suppose that we start with the opaque background color

(R0,G0,B0,1) – This color becomes the initial destination color • We now want to blend in a translucent polygon with color

(R1,G1,B1,α1) • Select GL_SRC_ALPHA and GL_ONE_MINUS_SRC_ALPHA as the source and destination blending factors ’ R 1 = α1 R1 +(1- α1) R0, …… • Note this formula is correct if polygon is either opaque or transparent

Angel: Interactive Computer Graphics 3E 43 © Addison-Wesley 2002 Clamping and Accuracy

• All the components (RGBA) are clamped and stay in the range (0,1) • However, in a typical system, RGBA values are only stored to 8 bits – Can easily loose accuracy if we add many components together – Example: add together n images • Divide all color components by n to avoid clamping • Blend with source factor = 1, destination factor = 1 • But division by n loses bits

Angel: Interactive Computer Graphics 3E 44 © Addison-Wesley 2002 Order Dependency

• Is this image correct? – Probably not – Polygons are rendered in the order they pass down the pipeline – Blending functions are order dependent

Angel: Interactive Computer Graphics 3E 45 © Addison-Wesley 2002 Opaque and Translucent Polygons

• Suppose that we have a group of polygons some of which are opaque and some translucent • How do we use hidden-surface removal? • Opaque polygons block all polygons behind them and affect the depth buffer • Translucent polygons should not affect depth buffer – Render with glDepthMask(GL_FALSE) which makes depth buffer read-only • Sort polygons first to remove order dependency

Angel: Interactive Computer Graphics 3E 46 © Addison-Wesley 2002 Fog

• We can composite with a fixed color and have the blending factors depend on depth – Simulates a fog effect

• Blend source color Cs and fog color Cf by Cs’=f Cs + (1-f) Cf • f is the fog factor – Exponential – Gaussian – Linear (depth cueing)

Angel: Interactive Computer Graphics 3E 47 © Addison-Wesley 2002 Fog Functions

Angel: Interactive Computer Graphics 3E 48 © Addison-Wesley 2002 OpenGL Fog Functions

GLfloat fcolor[4] = {……}: glEnable(GL_FOG); // fog function glFogf(GL_FOG_MODE, GL_EXP); // fog function parameter glFogf(GL_FOG_DENSITY, 0.5); glFogv(GL_FOG, fcolor); // near distance to start fog glFogf(GL_FOG_START, 1.5); // far distance to stop fog glFogf(GL_FOG_END, 10.5);

Angel: Interactive Computer Graphics 3E 49 © Addison-Wesley 2002 Fog Effect

http://www.engin.swarthmore.edu/~jshin1 Angel: Interactive Computer Graphics 3E 50 © Addison-Wesley 2002 Line Aliasing

• Ideal raster line is one pixel wide • All line segments, other than vertical and horizontal segments, partially cover pixels • Simple algorithms color only whole pixels • Lead to the “jaggies” or aliasing • Similar issue for polygons

Angel: Interactive Computer Graphics 3E 51 © Addison-Wesley 2002 Antialiasing by Area Averaging • Pixel shade is proportional to percentage of pixel covered by ideal line

Angel: Interactive Computer Graphics 3E 52 © Addison-Wesley 2002 Antialiasing

• Can try to color a pixel by adding a fraction of its color to the frame buffer – Fraction depends on percentage of pixel covered by fragment – Fraction depends on whether there is overlap

no overlap overlap

Angel: Interactive Computer Graphics 3E 53 © Addison-Wesley 2002 Area Averaging

• Use average area α1+α2-α1α2 as blending factor

Angel: Interactive Computer Graphics 3E 54 © Addison-Wesley 2002 OpenGL Antialiasing

• Can enable separately for points, lines, or polygons glEnable(GL_POINT_SMOOTH); glEnable(GL_LINE_SMOOTH); glEnable(GL_POLYGON_SMOOTH); • Pixels along edges are assigned fractional alpha values

glEnable(GL_BLEND); glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

• Compositing and blending are limited by resolution of the frame buffer – Typically 8 bits per color component • The accumulation buffer is a high resolution buffer (16 or more bits per component) that avoids loss of precision from multiple ops • Write into it or read from it with a scale factor • Slower than direct compositing into the frame buffer • glAccum(Glenum operation, Glfloat value);

• glDrawBuffer(Glenum mode); • Specifies up to four color buffers to be drawn into • mode – GL_NONE, GL_FRONT_LEFT, GL_FRONT_RIGHT, GL_BACK_LEFT, GL_BACK_RIGHT, GL_FRONT, GL_BACK, GL_LEFT, GL_RIGHT, GL_FRONT_AND_BACK, GL_AUXi

• glAccum(Glenum operation, Glfloat value); • Operation – GL_ACCUM : Multiplies by value then adds the RGBA values from the buffer selected for reading by glReadBuffer to the accumulation buffer – GL_LOAD : Multiplies by value then copies the RGBA values from the buffer selected for reading by glReadBuffer to the accumulation buffer – GL_ADD : Adds value to each RGBA in the accumulation buffer – GL_MULT : Multiplies each RGBA in the accumulation buffer by value – GL_RETURN : Transfers accumulation buffer values to the color buffer(s) currently selected for writing by glDrawBuffer

• Compositing • Image Filtering (convolution) • Whole scene antialiasing • Motion effects • Depth-of-field effects

• glDrawBuffer(GL_AUX0); • glReadBuffer(GL_AUX0); • Loop n time – Jitter image plane & draw • Less than one pixel – glAccum(GL_ACCUM, 1.0/n); • glDrawBuffer(GL_BACK); • glAccum(GL_RETURN, 1.0);

• Jitter camera • Each frustum has common plane “in focus” • Accumulate images

Angel: Interactive Computer Graphics 3E 62 http://www.cs.stevens.edu/~quynh © Addison-Wesley 2002 OpenGL Perspective glFrustum(xmin,xmax,ymin,ymax,near,far)

-near)

• With glFrustum it is often difficult to get the desired view • gluPerpective(fovy, aspect, near, far) often provides a better interface front plane

aspect = w/h

• glFrustum allows for an unsymmetric viewing frustum • gluPerspective does not -zmaxmax

-zmin) -zmin)

• Specifies which pixels to draw • Create arbitrary shaped viewports • glStencilFunc()- sets comparison function • glStencilOp() sets stencil operations

• One of the most noticeable effect of inter-object lighting • Direct calculation of the physics (ray tracing) is too expensive • Our focus is to capture the most significant reflection while minimizing the overhead via rendering the “virtual object”

Clipped to reflector virtual object

reflector

object

• Image space methods: create a texture from a view of the reflected objects and apply it to the reflector – Advantage: does not depend on the object geometry – Disadvantage: sampling issue and also only an approximation (environment mapping as an example) • Object space methods: create the actual geometry of the object to be reflected and render it to the reflector – Disadvantage: accuracy of the geometry – Advantage: more accurate reflection (for nearby objects) • Both methods need to create the virtual objects

• The most common reflection – flat mirror, floor, wall, etc • Creating virtual objects (or reflected objects) is much easier • A view independent operation – only consider the relative position of the object and the reflector • The virtual object is created by transforming the object across the reflector plan

• An important task: clip the reflected geometry so it is only visible on the reflector surface – Beyond the reflector boundaries and in front of reflector

side front

• The key is you only want the reflected geometry to appear on the reflector surface • Use stencil buffer: – Clear the stencil buffer – Render the reflector and set the stencil – Render the reflected geometry only where the stencil pixels have been set • The above algorithm is to use the stencil buffer to control where to draw the reflection

• Another method: render the reflected object first, and then render the reflector to set the stencil buffer, then clear the color buffer everywhere except where the stencil is set

• This method is to use the stencil buffer to control where to erase the incorrect reflection

• Advantage: when it is faster to use stencil to control clearing the scene than drawing the entire scene with stencil tests