To Do

Advanced § Assignment 2 due Feb 19 § Should already be well on way. CSE 190 [Winter 2016], Lecture 12 § Contact us for difficulties etc. Ravi Ramamoorthi http://www.cs.ucsd.edu/~ravir

Motivation Real-Time Rendering

§ Today, create photorealistic computer graphics § Goal: interactive rendering. Critical in many apps § Complex geometry, , materials, shadows § Games, visualization, computer-aided design, … § Computer-generated movies/special effects (difficult or impossible to tell real from rendered…) § Until 10-15 years ago, focus on complex geometry

§ CSE 168 images from rendering competition (2011) § § But algorithms are very slow (hours to days) Chasm between interactivity, realism

Evolution of 3D graphics rendering Offline 3D Graphics Rendering

Interactive 3D graphics as in OpenGL , radiosity, photon mapping § Earliest SGI machines (Clark 82) to today § High realism (global illum, shadows, refraction, lighting,..) § Most of focus on more geometry, § But historically very slow techniques § Some tweaks for realism (, accum. buffer) “So, while you and your children’s children are waiting for ray tracing to take over the world, what do you do in the meantime?” Real-Time Rendering

SGI Reality Engine 93 (Kurt Akeley) Pictures courtesy Henrik Wann Jensen

1 New Trend: Acquired Data 15 years ago

§ Image-Based Rendering: Real/precomputed images as input § High quality rendering: ray tracing, § Little change in CSE 168 syllabus, from 2003 to today § Also, acquire geometry, lighting, materials from real world § Real-Time rendering: Interactive 3D geometry with simple § Easy to obtain or precompute lots of high quality data. But texture mapping, fake shadows (OpenGL, DirectX) how do we represent and reuse this for (real-time) rendering? § Complex environment lighting, real materials (velvet, satin, paints), soft shadows, caustics often omitted in both

§ Realism, interactivity at cross purposes

Today: Real-Time Game Renderings Today

§ Vast increase in CPU power, modern instrs (SSE, Multi-Core) § Real-time raytracing techniques are possible (even on hardware: Optix)

Unreal Engine 4 https://www.youtube.com/watch?v=gtHamLNPXyk#t=33 § 4th generation of graphics hardware is programmable § (First 3 gens were wireframe, shaded, textured) § Modern NVIDIA, ATI cards allow vertex, fragment

§ Great deal of current work on acquiring and rendering with realistic lighting, materials… [Especially at UCSD] § Focus on quality of rendering, not quantity of polygons, texture Digital Ira: NVIDIA, USC

Goals Outline

§ Overview of basic techniques for high-quality § Motivation and Demos real-time rendering § Programmable § Shadow Maps § Survey of important concepts and ideas, but do § Environment Mapping not go into details of writing code

§ Some pointers to resources, others on web § One possibility for assignment 3, will need to think about some ideas on your own

2 High quality real-time rendering High quality real-time rendering § Photorealism, not just more polygons § Photorealism, not just more polygons § Natural lighting, materials, shadows § Natural lighting, materials, shadows

Glass Star (courtesy Intel) Peacock feather Glass Vase Real materials diverse and not easy to represent by simple Interiors by architect Frank Gehry. Note rich lighting, ranging parameteric models. Want to support measured reflectance. from localized sources to reflections off vast sheets of glass.

High quality real-time rendering Today: Full Global Illumination § Photorealism, not just more polygons § Natural lighting, materials, shadows

small area , sharp shadows soft and hard shadows Agrawala et al. 00 Ng et al. 03 Natural lighting creates a mix of soft diffuse and hard shadows.

Applications Programmable Graphics Hardware

§ Entertainment: Lighting design § Architectural visualization § Material design: Automobile industry § Realistic Video games § Electronic commerce

3 Programmable Graphics Hardware Precomputation-Based Methods

§ Static geometry

§ Precomputation

§ Real-Time Rendering (relight all-frequency effects)

NVIDIA a new dawn demo http://www.geforce.com/games-applications/pc- § Involves sophisticated representations, algorithms applications/a-new-dawn/videos

Relit Images Video: Real-Time Relighting

Ng, Ramamoorthi, Hanrahan 04

Spherical Harmonic Lighting Interactive RayTracing

Advantages § Very complex scenes relatively easy (hierarchical bbox) § Complex materials and for free § Easy to add global illumination, specularities etc. Disadvantages § Hard to access data in memory-coherent way § Many samples for complex lighting and materials § Global illumination possible but expensive Modern developments: Leverage power of modern CPUs, develop -aware, parallel implementations http://www.geforce.com/games-applications/pc-applications/ design-garage/videos Avatar 2010, based on Ramamoorthi and Hanrahan 01, Sloan 02

4 Sparse Sampling, Reconstruction Sparse Sampling, Reconstruction

§ Same algorithm as offline Monte Carlo rendering § But with smart sampling and filtering (current work)

Outline Basic Hardware Pipeline

§ Motivation and Demos Application Geometry Rasterizer

§ Programmable Graphics Pipeline CPU GPU § Shadow Maps § Environment Mapping

Create geometry, , Transform and lighting calcs. materials, textures, Apply per-vertex operations Textures, Cubemaps cubemaps, … as inputs Per-pixel (per-fragment) operations

Geometry or Pixel or Fragment Pipeline

Model, View Rasterization Texture Lighting Projection Clipping Screen Z-buffering Transform (scan conversion) Mapping

These fixed function stages can be replaced by a general per-vertex These fixed function stages can be replaced by a general per-fragment calculation using vertex shaders in modern programmable hardware calculation using fragment shaders in modern programmable hardware

5 OpenGL Rendering Pipeline Simplified OpenGL Pipeline Programmable in § User specifies vertices (vertex buffer object) Modern GPUs Programmable in (Vertex ) Modern GPUs § For each vertex in parallel Geometry (Fragment § OpenGL calls user-specified vertex shader: Vertices Primitive Shader) Transform vertex (ModelView, Projection), other ops

Scan Framebuffer Operations Fragment Conversion Operations § For each primitive, OpenGL rasterizes (Rasterize) § Generates a fragment for each pixel the fragment covers § For each fragment in parallel Pixel Texture Images § OpenGL calls user-specified fragment shader: Operations Memory Shading and lighting calculations § OpenGL handles z-buffer depth test unless overwritten Traditional Approach: Fixed function pipeline (state machine) § Modern OpenGL is “lite” basically just a rasterizer New Development (2003-): Programmable pipeline § “Real” action in user-defined vertex, fragment shaders

Shading Languages Shader Setup

§ Initializing (shader itself discussed later) § Vertex / Fragment shading described by small program § Written in language similar to C but with restrictions 1. Create shader (Vertex and Fragment) § Long history. Cook’s paper on Shade Trees, 2. Compile shader Renderman for offline rendering 3. Attach shader to program 4. Link program § Stanford Real-Time , work at SGI § Cg from NVIDIA, HLSL 5. Use program § GLSL directly compatible with OpenGL 2.0 (So, you can § Shader source is just sequence of strings just read the OpenGL Red Book to get started) § Similar steps to compile a normal program

Shader Initialization Code Linking Shader Program

GLuint initshaders (GLenum type, const char *filename) { GLuint initprogram (GLuint vertexshader, GLuint fragmentshader) // Using GLSL shaders, OpenGL book, page 679 { GLuint shader = glCreateShader(type) ; GLuint program = glCreateProgram() ; GLint compiled ; GLint linked ; string str = textFileRead (filename) ; glAttachShader(program, vertexshader) ; GLchar * cstr = new GLchar[str.size()+1] ; glAttachShader(program, fragmentshader) ; const GLchar * cstr2 = cstr ; // Weirdness to get a const char glLinkProgram(program) ; strcpy(cstr,str.c_str()) ; glGetProgramiv(program, GL_LINK_STATUS, &linked) ; glShaderSource (shader, 1, &cstr2, NULL) ; if (linked) glUseProgram(program) ; glCompileShader (shader) ; else { glGetShaderiv (shader, GL_COMPILE_STATUS, &compiled) ; programerrors(program) ; if (!compiled) { throw 4 ; shadererrors (shader) ; } throw 3 ; return program ; } } return shader ; }

6 Cliff Lindsay web.cs.wpi.edu/~rich/courses/imgd4000-d09/lectures/gpu.pdf Cliff Lindsay web.cs.wpi.edu/~rich/courses/imgd4000-d09/lectures/gpu.pdf

Fragment Shader Compute Lighting Outline

vec4 ComputeLight (const in vec3 direction, const in vec4 lightcolor, const in vec3 normal, const in vec3 halfvec, const § Motivation and Demos in vec4 mydiffuse, const in vec4 myspecular, const in float myshininess) { § Programmable Graphics Pipeline float nDotL = dot(normal, direction) ; § vec4 lambert = mydiffuse * lightcolor * max (nDotL, 0.0) ; Shadow Maps § float nDotH = dot(normal, halfvec) ; Environment Mapping vec4 phong = myspecular * lightcolor * pow (max(nDotH, 0.0), myshininess) ;

vec4 retval = lambert + phong ; return retval ; }

Shadow and Environment Maps Common Real-time Shadow Techniques § Basic methods to add realism to interactive rendering Projected Shadow § Shadow maps: image-based way hard shadows planar volumes § Very old technique. Originally Williams 78 shadows § Many recent (and older) extensions § Widely used even in (RenderMan) § Simple alternative to raytracing for shadows § Environment maps: image-based complex lighting § Again, very old technique. Blinn and Newell 76 § Hybrid Huge amount of recent work (some covered in course) approaches § Together, give most of realistic effects we want § But cannot be easily combined!! § See Annen 08 [real-time all-frequency shadows dynamic scenes] for one approach: convolution soft shadows Light maps This slide, others courtesy Mark Kilgard

7 Problems Shadow Mapping

Mostly tricks with lots of limitations § Lance Williams: Brute Force in image space (shadow maps in 1978, but other similar ideas like § Projected planar shadows Z buffer, using textures and so on) works well only on flat surfaces

§ Stenciled shadow volumes § Completely image-space algorithm determining the shadow volume is hard work § no knowledge of scene’s geometry is required § must deal with aliasing artifacts § Light maps totally unsuited for dynamic shadows § Well known software rendering technique § In general, hard to get everything shadowing everything § Basic shadowing technique for Toy Story, etc.

Phase 1: Render from Light Phase 1: Render from Light

§ Depth image from light source § Depth image from light source

Phase 2: Render from Eye Phase 2+: Project to light for shadows

§ Standard image (with depth) from eye § Project visible points in eye view back to light source

Eye Eye

(Reprojected) depths match for light and eye. VISIBLE

8 Phase 2+: Project to light for shadows Visualizing Shadow Mapping

§ Project visible points in eye view back to light source § A fairly complex scene with shadows

the point Eye light source

(Reprojected) depths from light, eye not the same. BLOCKED!!

Visualizing Shadow Mapping Visualizing Shadow Mapping

§ Compare with and without shadows § The scene from the light’s point-of-view

FYI: from the eye’s point-of-view again

with shadows without shadows

Visualizing Shadow Mapping Visualizing Shadow Mapping

§ The depth buffer from the light’s point-of-view § Projecting the depth map onto the eye’s view

FYI: from the FYI: depth map for light’s point-of-view light’s point-of-view again again

9 Visualizing Shadow Mapping Visualizing Shadow Mapping

§ Comparing light distance to light depth map § Scene with shadows

Green is where the light planar Notice how Notice how Non-green is distance and specular curved where the light highlights surfaces cast shadows depth map never appear shadows on should be are in shadows each other approximatel y equal

Hardware Shadow Map Filtering Hardware Shadow Map Filtering

GL_NEAREST: blocky GL_LINEAR: antialiased edges “Percentage Closer” filtering § Normal texture filtering just averages components § Averaging depth values does NOT work § Solution [Reeves, SIGGARPH 87] § Hardware performs comparison for each sample § Then, averages results of comparisons § Provides anti-aliasing at shadow map edges § Not soft shadows in the umbra/penumbra sense

Low shadow map resolution used to heighten filtering artifacts

Problems with shadow maps Reflection Maps

§ Hard shadows (point lights only) § Quality depends on shadow map resolution (general problem with image-based techniques) § Involves equality comparison of floating point depth values means issues of scale, bias, tolerance § Some of these addressed in papers presented

Blinn and Newell, 1976

1 0 Environment Maps Environment Maps

Miller and Hoffman, 1984 Interface, Chou and Williams (ca. 1985)

Environment Maps Reflectance Maps

§ Reflectance Maps (Index by N) § Horn, 1977 Cylindrical Panoramas § Irradiance (N) and Phong (R) Reflection Maps § Miller and Hoffman, 1984

180 degree fisheye Mirror Sphere Chrome Sphere Matte Sphere Cubical Environment Map Photo by R. Packo

Irradiance Environment Maps Assumptions R N § Diffuse surfaces § Distant illumination § No shadowing, interreflection

Hence, Irradiance a function of surface normal

Incident Radiance Irradiance Environment Map (Illumination Environment Map)

1 1 Diffuse Reflection Analytic Irradiance Formula BE= ρ π Lambertian surface 2π /3 Radiosity Reflectance Irradiance acts like low-pass filter (image intensity) (albedo/texture) (incoming light) Al π /4 E = A L lm l lm 0 0 1 2 = × l

l −1 Ramamoorthi and Hanrahan 01 2 ⎡ ⎤ (−1) ⎢ l! ⎥ Al = 2π 2 l even Basri and Jacobs 01 (l + 2)(l −1) ⎢ l l ⎥ quake light map 2 2 ! ⎣ ( ) ⎦

9 Parameter Approximation 9 Parameter Approximation

Order 0 Order 1 Exact image Exact image 1 term 4 terms

m m Y (θ,ϕ) Y (θ,ϕ) 0 lm 0 lm RMS error = 25 % l RMS Error = 8% l

1 z 1 y x y z x

2 yz 2 2 2 2 yz 2 2 2 xy 3z −1 zx x − y xy 3z −1 zx x − y -2 -1 0 1 2 -2 -1 0 1 2

9 Parameter Approximation Real-Time Rendering E(n) = ntMn Order 2 Simple procedural rendering method (no textures) Exact image 9 terms § Requires only matrix-vector multiply and dot-product § In software or NVIDIA vertex programming hardware Widely used in Games (AMPED for Microsoft

m Xbox), Movies (, Framestore CFC, …) Y (θ,ϕ) 0 lm RMS Error = 1% l surface float1 irradmat (matrix4 M, float3 v) { 1 For any illumination, average y z x float4 n = {v , 1} ;

error < 3% [Basri Jacobs 01] 2 yz 2 2 2 return dot(n , M*n) ; xy 3z −1 zx x − y

-2 -1 0 1 2 }

1 2 Environment Map Summary Resources

§ Very popular for interactive rendering § OpenGL red book (latest includes GLSL) § Web tutorials: http://www.lighthouse3d.com/opengl/glsl/ § Extensions handle complex materials § Older books: OpenGL Shading Language book (Rost), § Shadows with precomputed transfer The Cg Tutorial, … § http://www.realtimerendering.com § Real-Time Rendering by Moller and Haines § But cannot directly combine with shadow maps § Debevec http://www.debevec.org/ReflectionMapping/ § Links to Miller and Hoffman original, Haeberli/Segal § Limited to distant lighting assumption § http://www.cs.berkeley.edu/~ravir/papers/envmap § Also papers by Heidrich, Cabral, … § Lots of information available on web… § Look at resources from CSE 291 website

1 3