Interactive Graphical Systems Fall 2002
Historical Overview
1960 First random scanned display available 1
Interactive Graphical Systems 1964 A few applications using computer graphics ($100.000) 100
Fall 2002 1968 Phosphor storage tubes increasingly available ($5000) 1000
1978 Raster scanned displays become increasingly popular 100.000
1984 First PC with graphical user interface (Macintosh) 1000.000 Image Generators and Displays Stefan Seipel 1994 Graphical user interfaces replaces increasingly text interface 50.000.000 1999 Almost no computer available without GUI 200.000.000
(Source: E.Bengtsson, Uppsala University)
Random scan display Random scan display - computer control
• based on phosphor storage tube • computer keeps a list of graphical primitives • cathode ray is controlled arbitrarily in XY direction • analog control of ray implemented in electronic circuits • computer issues rendering and parameters • sets of vector primitives available (points, lines ) • very little memory required to store display list • very smooth lines / high resolution • no management of graphics buffer required • line drawing only • no filled surfaces • display/engine and computer strictly separated • no block transfers • limited time frame to draw graphics • up to 100.000 short vectors per refresh cycle Raster scan display Raster scan display - computer control
• based on TV technology - short glowing phosphors • computer manipulates values in the frame buffer
• screen is scanned in a line-by-line order • discrete frame buffer matrix -> rasterization
• pixels within a line are switched on/off while line is scanned • arbitrary graphical objects can theoretically been displayed
• two-dimensional memory matrix (frame buffer) contains pixel value • need for rasterization algorithms
• filled areas can be drawn • graphical primitives must be computed
• screen must be refreshed in short intervals • host computer calculates and accesses frame-buffer
• memory must be read out -> video signal must be generated
Frame buffer readout - RAMDAC Simple Raster Scan Display Architecture
Frame buffer RAM (random access memory) CPU Video Controller other periphery
create video DAC signal host memory memory (digital to analog converter) write read System Bus
(frame buffer operations)
• readout of 1280x1024 pixels at 60 Hz -> memory access time 12.7 nanoseconds !
System Video Video Monitor • “normal” RAM have an average access time of 60ns Memory Memory Controller
• special purpose frame buffer required
• must allow for simultaneous read and write operations More or less typical configuration with common graphics cards Typical frame buffer operations: 2D graphics subsystem
CPU other periphery 2D - Graphics operations
• BitBlt (bit block transfer) System Bus •Draw Pixel Frame buffer operations are a heavy burden for (2D graphical primitives) •Draw Line a single CPU system !
•Draw Text System Raster Video Monitor Memory Engine Controller • Fill Area Frame (color) Buffer
2D graphics subsystem
Color buffer standards (common resolutions) Memory requirements and signal bandwidths
Spatial resolutions: CGA - 320x200 Spatial Resolution SXGA 1280x1024 EGA - 640x350 Color Resolution True color 24 bit VGA - 640x480 SVGA - 800x600 Screen Refresh 72 Hz XGA - 1024x768 SXGA - 1280x1024 or 1600x1280 Video Memory Requirement : 1280*1024*24 bit = 3.75 Mbyte (1.3Mio pixel) HDTV - 1900x1200 ATC - 2000 x 2000 Human eye - 10.000 x 10.000 (not evenly distributed) Video readout : 94 Mio./pixel per sec.
Color resolutions: Memory readout : 226 Mio. bits per sec. ! palletized - 256 colors lookup-table (8 bit/pixel) grayscale - 256 shades (8 bit /pixel) -> very special memory readout required grayscale - 4096 shades (12 bit /pixel) e.g. in medical apps. Hi color - 5:5:5 RGB, or 5:6:5 RGB (15 or 16 bit/pixel) (e.g. 240 MHz RAMDAC) true color - 8:8:8 RGB (24 bit/pixel) true color - 8:8:8:8 RGBA (32 bit/pixel) The standard 3D graphics pipeline
Modeling Viewing Display Rasteri- 3D graphics adds another dimension Transfor- Lightning Transfor- Clipping Projection List zation mation mation Traversal 2D color buffer only is not sufficient Geometric processing Rasterization - thousands of flops - operates on pixels (framebuffer) - vertex transform - millions of iops - normal transforms - alpha compares - lightning calculation - depth buffer test - stencil test - alpha arithmetic - texture addressing
Features of 3D image generators Features of 3D image generators
Geometric processing (per vertex operations): Rasterization (per pixel operations): One or several raster engines feature: • one or several floating point geometry engines • discrete line drawing and polygon fill • perform matrix and vector operations • z-Buffer test (depth buffer test) • z-Buffer blending (fog) hardware supported graphics operations: • transformation of vertices • blending with alpha-channel (transparency) • rotation of vectors • color interpolation (e.g. Gouraud shading) • normalization of vectors (after scale of object) • texel addressing • calculation of lightning • tri/bilinear interpolation of textures • projection & clipping • anti-aliasing of edges • stencil test Advanced 3D graphics subsystem Buffers configuration example (3D system)
double buffer CPU (front and back) other periphery
accumulation buffer 8 b System Bus
depth buffer 24 / 32 bit
alpha buffer 8 bit blue-color buffer 8 bit green-color buffer 8 bit red-color buffer 8 bit
Geometry Raster Video System Monitor Memory Processor(s) Engine(s) Controller stencil and/or overlay buffer 8 bit Frame Buffer(s) Local Memory Other Buffers (caches, textures) (Stencil,Depth,Alpha…) Memory required per pixel: 2*24+8+8+32+8 = 128 bit = 16 byte
Graphics subsystem For a resolution of 1280x1024 -> 20 MB frame buffer
Buffers configuration example (3D stereo system) Depth-Buffer Aliasing
left right
Fixed number of bits in the z-Buffer limits front resolution of the scene depth Quad- accumulation buffer 8 bit buffer Example: depth buffer 24 / 32 bit 16 bit depth buffer allows only 65536 discrete steps in depth back alpha buffer 8 bit blue-color buffer 8 bit green-color buffer 8 bit Arithmetical rounding operation red-color buffer 8 bit of floating point depth values causes ambiguous z-values !
stencil and/or overlay buffer 8 bit Visual artifacts in rendering of objects which are very close to Memory required per pixel: 4*24+8+8+32+8 = 176 bit = 22 byte each other (see picture) For a resolution of 1280x1024 -> 27,5 MB frame buffer Performance parameters – History (1999) Performance parameters - What do they say ?
Pixel Fillrate: Problem: • refers to rasterization performance • number of shaded/textured/buffered pixels per second • stated performance values are often only achieved under specific circumstances • common: 5-40 million/sec. (low and medium cost game accelerators) • often these values refer to the most simple rendering modes (no shading, no z-buffer) • quite good: 100 million/sec. (graphics workstations) • performance is often achieved with native implementations • high end: 500-2000 million/sec. (e.g. SGI top of the line) • quality of driver implementation is essential Geometry Performance: • comparability is a tricky issue, since various new technical features • refers to throughput of graphical primitives • number of 3D shaded triangles/sec. (of certain size e.g. 25 pixel) Therefore: • number of 3D shaded lines/sec. (of certain length e.g. 10 pixels) • don´t trust vendor supplied specifications ! • common: <0.5 million triangles/sec. (low cost game accelerators) • if you are lucky, you reach 10%-20% of the stated performance ! • quite good: <1-2 million triangles/sec. (graphics workstations) • graphics sub-systems should be tried before bought ! • high end: <50 million/sec. (e.g. SGI Octane) • in the target system (depending largely on CPU configuration) • using your application (depending on typical graphics operations) Nowadays (2002) performance doubled with ten even on PC hardware The gap between professional systems and consumer products is closing • Always check : is there driver support ? are there drivers at all ?
Bottlenecks in 3D graphics systems Example bottleneck evaluation
Geometry bound systems User requirements for a flight simulation: - too many polygons in the scene • terrain model 1000 textured polygons, will fill most of the screen background • airplane model 5000 polygons, average size 200 pixels - too complex lightning calculation • for quality reasons, rendering should appear in SXGA resolution - too complex model transformations • frame-rate >40/sec.
Fill bound systems There is a graphics subsystem available: - too big areas to fill (number of pixels per polygon) • 60 Mio. textured, lit, shaded pixels/sec. - too many large polygons • 2.000.000 triangles/sec. (25 pixels) - high overdraw ratio • cost: 30.000 SEK - too many textures in scene - too complex alpha arithmetic (blending, fog) Is it advisable to buy this graphics subsystem ? Fill bound! • (1280x1024+5000x200)x40 = 92.428.800 pixels/sec. Depends on your application and graphics system • (1000+5000)x2x40 = 480.000 triangles/sec API´s (application programmers interface) API´s continued
Avoid to develop hardware oriented software Different standard 3D API´s:
Minimize turnaround costs and time Many different available today. The most renowned are:
Use well established and standardized 3D API´s PHIGS and GKS (old DEC machines, not common any longer) OpenGL (SGI, Microsoft, many others) Use graphics accelerators which support those API´s QuickDraw 3D (Apple) - full functional support Direct3D (Microsoft) - optimized performance for these drivers Glide (3Dfx) Heidi (used by many CAD programs) The best graphics hardware is worth nothing without Quake (sort of standard for games) appropriate API and driver support !
Physiological Aspects Displays
Spatial Retinal Resolution : 1´
Additional Reading Visual Field : approx. 200o, with 120o binocular overlap
Roy S. Kalawsky: The Science of Virtual Reality and Virtual Environments Addison-Wesley Publishing Company, 1993, ISBN 0-201-63171-7 Limits of depth perception from lateral disparity Perception: pages 50-59 Displays techniques: pages 98-107 Temporal Resolution : approx. 50 Hz, increasing with luminance Visual Displays - Basic Technologies Basic Technologies - Cathode Ray Tubes (CRT)
• Cathode Ray Tubes Acceleration Anode • Flat Panel Displays Advantages: Cathode • Electroluminiscence Displays - high resolution •LCD Displays - easy to control • Active Matrix TFT - reliable technology • Light Valves -low cost • Laser Scanners Focussing Electron gun x-/y- •Micro Mirror Devices Deflection (elstat / magn.)
Phosphor Coating
Basic Technologies - Cathode Ray Tubes (CRT) Basic Technologies - Random Scan CRT
Operation modes of CRTs: How it works: Display primitives are stored in a display list (ellipses, circles, lines...)
Electron beam is controlled continuously between vertice -> analog line drawing Random scan mode Raster scan mode
+ no aliasing, smoth lines + transformation in hardware possible + little memory required
- only line drawing possible - complex control hardware for the beam - flickers if scene becomes complex Basic Technologies - Raster Scan CRT Basic Technologies - Raster Scan CRT
How it works: Frame buffer
Display is rasterized into pixels which are stored 00000000011100000000000000100010 00000000101010000001010100011000 00000000000000000100100011111000 in a two-dimensional memory array (raster memory) 00000000000000000010010100001001 Host 00000000000010101000100111111000 Host 00000000000001011001111001011000 00000000000000000000101001010011 Electron beam is traversing the screen line by line in a regular time frame and scheme VideoVideo Content of the raster memory controls beam intensity DisplayDisplay Controller Controller ControllerController scan line Converts analog primitive + uses old TV technology (simple approach) into discrete representation Converts discrete frame buffer + filled and shaded surfaces are possible into analog video signal + guaranteed screen refresh rate horizont retrace - aliasing problems “staircase effect” - electronics required for raster memory readout vertica - drawing algorithms more complex retrace
Basic Technologies - Cathode Ray Tubes (CRT) Basic Technologies - Cathode Ray Tubes (CRT)
How is color accomplished in displays ? The role of the phosphor coating : - Fluorescence (glowing when hit by electron beam) Most usually, colors are mixed by additive composition of base colors - Phosphorescence (after glowing while being activated) - Persistency (time until glowing phosphorescence decreases below 10%) typically 5-60 milliseconds. 1. Spatially modulated color composition
- Persistency is important. Short persistency requires high update rates otherwise see page 99, in Kalawsky flicker. Long persistency causes stabile but smeary images.
- Granularity of the phosphor -> spot size, image resolution 2. Temporally modulated color composition
- Type of phosphor defines color: see next slide p1 : green, average persistency p4 : white, short persistency p12 : orange, average persistency p31 : green, short persistency Basic Technologies - Cathode Ray Tubes (CRT) Parameters for display assessment
How is color accomplished ? - dot size (mm) - dot pitch (mm) Color shutter technology. - resolution (lp/mm) - Sequential display of color fields on monochromatic CRT -brightness (cd/m2) - Synchronization of the fields with color filters - contrast ratio 1mm 1mm - Filters can be : -display size a) LCD filters (electronically controlled) - addressability 1mm 1mm b) Optical filters (mechanically coupled) - refresh rate dot size ? dot pitch? - color range resolution? - Advantages : No color convergence errors - convergence - Disadvantages : High frequent oscillations in the visual field decompose colors - weight / power consumption
Basic Technologies - Typical parameters for CRTs Basic Technologies - Flat Panel Display - LCD
Liquid Crystal Displays
Screen Diagonal Size (14”-26”)
Shadow masks (Triple holes, Strips) vert. pol.filter
Dot pitch (0.24 - 0.30 mm) vertical electrodes Video bandwidth (50 -250 MHz) Horizontal Sync. Frequency (30 - 170 kHz) liquid crystal layer
Vertical Sync. Frequency ( 48 - 170 Hz) horizontal transparent max. Resolution (1280x1024 - 4800 x 4000) electrodes hor.pol.filter
Light source or mirror No electric field = light passes through Basic Technologies - Flat Panel Display - LCD Basic Technologies - Flat Panel Display
Addressing of Pixels: Thin Film Transistor Matrix (TFT):
Deposit an electronic charge on intersections between horizontal and vertical electrodes sequentially row by row. An array of transparent transistors is deposited on the LCD Limited speed, since certain minimum time is required to is deposited (depending on the capacity of the intersections) Pixels can be switched on and off
When last row has been addressed, first rows have already lost Pixel keep their electrical state and their electric charge optical properties Poor contrast image => Significant contrast and intensity enhancements
Basic Technologies - Flat Panel Display Basic Technologies - Flat Panel Display
Color reproduction / Optical Efficiency: Electro Luminescence Displays (plasma panel displays):
Groups of adjacent pixels are forming one effective color pixel Gas is encapsulated between electrodes Potential
Sub-pixels are covered with color filters When a certain amount of voltage is applied S Strike Voltage (striking voltage) the plasma discharges and Common sub-pixels configuration are RGB stripes, triads or quads glows until the potential drops below the discharge voltage. D Efficient resolution is reduced Plasma cell keeps luminating for a while Discharge Voltage without being refreshed.
Light intensity is diminished significantly when passing through Active luminance, high intensity display polarizes, liquid crystals, and color filters (poor optical efficiency) Plasma generates light t (see also page 96) Basic Technologies - Flat Panel Display Basic Technologies - Light Valves
Electro Luminescence Displays (plasma panel displays): Application : Projection Displays
Sandwich-Technique High resolution light modulation (>1600 x 1200) glas High refresh rates possible (>130 Hz) vertical transparent electrodes Usable for high light output projection displays glass substrate with plasma cells (Neon, Argon) The first choice for stereo projection systems horizontal transparent electrodes Tricky problems : Ghost images with slow phosphor glass
see page 103
Basic Technologies - Laser Scanner Basic Technologies - Micromirror Devices (MMD)
Used for: Screen Matrix of micro mirrors large screen projection displays Addressable and electronically controllable direct retinal displays Used for Light Reflection and Projection Display Systems (very high resolution) Extremely high optical efficiency for color displays, several lasers required (convergence ?) vertical horizontal deflection deflection mirror mirror
laser source Basic Technologies - Micro Mirror Devices (MMD) Visual Displays - 3D Displays and Optical Systems
3D Projection Systems MMD System Working Principle Front Projection Rear / Retro Projection
Autostereoscopic Displays Slot Mask Lenticular / Double Lenticular Arrays
Volume Displays
Other Optical Coupled Displays Fiber coupled displays (HMD see chapter 4.2.1) Lens/Mirror coupled displays (HMD see chapter 4.2.1)
3D Displays and Optical Systems - Projection Systems 3D Displays and Optical Systems - Projection Systems
3D Projection Display Systems require: 3D Front Projection Systems: Image source and observer are located on the same side of the projection surface Very high intensity image source ( - user may interfere with projection beam) • Transmission LCD/TFT Panel + Light Source • Reflection LCD/TFT Panel + Light Source Stereo 3D with active shutter glasses • requires very fast image source (>120 Hz) • Projection CRT (specialized high intensity CRT tube ) • light valve / projection CRT • MMD • extremely expensive Screen • Light Valve • single graphics pipeline Image Source •(Laser) • screen with good diffuser properties + Optics Stereo 3D with passive polarizing glasses Focusing optics/color splitter • requires two image sources • Wide / narrow angle optics, fixed or variable • image alignment problems • dual graphics pipeline required • requires special silver screen which Projection screen preserves polarization • Transparency / Diffusion / Specular Properties Stereo 3D with color field separation (red/green) • one color capable projector (cheap) Observer Means of splitting left/right channel • poor image result • Time Multiplexing / Color Separation / Polarization • screen with good diffuser properties 3D Displays and Optical Systems - Projection Systems 3D Displays and Optical Systems - Projection Systems
3D Rear / Retro Projection Systems: 3D Rear / Retro Projection Systems: Examples Image source is positioned behind the projection screen Image Source No interference between user and image source + Optics
Requires transparent screen material Screen No polarized 3D stereo possible since polarization is disturbed in transmission
Stereo 3D with active shutter glasses • requires very fast image source • light valve / projection crt Observer Caves Virtual Planes Viewing wands • expensive • single graphics pipeline Attention has to paid to mirror effects
3D Displays / Optical Systems - Autostereoscopic Displays 3D Displays / Optical Systems - Autostereoscopic Displays
Autostereoscopy - stereoscopic perception with the “naked” eye (Double) Lenticular Lens Arrays LCD-Projektor LCD-Projektor Image Splitter (Sanyo)
• Display divided in vertical stripes • Display divided in vertical stripes • Alternate stripes display left and right image Pixel Column R Pixel Column R • Alternate stripes display left and right image Pixel Column L Pixel Column L Pixel Column R Pixel Column L • Half-Cylinder shaped lenses project the Display surface • Slit-mask is blocking out the view of the left stripes to the corresponding eye eye onto the right picture and vice versa Double-lenticular retro-projection system Slit mask • Several viewing zones • Only a single user • Dedicated observer distance • Dedicated observer position LCD/TFT Panel • Horizontal resolution decreased • Horizontal resolution decreased
Lenticular flat panel display system 3D Displays / Optical Systems - Autostereoscopic Displays 3D Displays / Optical Systems - Volumetric Displays
The display creates a real volumetric representation which is perceived as a 3D Examples (Heinrich-Hertz Institute, Berlin) structure without the need for glasses or other aids.
The idea is to project dynamic images onto oscillating or rotating surfaces in order to create the sensation of a volumetric object.
Prototypes have been build using: - rotating LED matrices - rotating helical projection surfaces with laser projection laser - lasers projecting into fog - experiments are underway to bring a solid crystal to illumination on addressable positions
• Allows for user movements All these systems can only show transparent/monochrome objects • Allows for user movements • Uses head-tracking • Uses head-tracking Mechanical problems and limits, dead viewing areas • Screen is automatically positioned • Lenticular array is shifted correctly with a robot arm Commercial systems are far ahead
3D Displays / Optical Systems - Volumetric Displays Choice of VR Displays - Evaluation of Requirements
How many observers are watching at the same time ?
What resolution and color fidelity requirements are there ? -> basic display technology
Is wide field of view desirable ?
Is immersion an important issue for the application ?
Is stereoscopic 3D rendering required ?
If yes, decide which type • one screen polarized -> take care for optical properties of the system • one screen time multiplexed -> display must tolerate high refresh rates • dual screen (HMD) -> check for resolution • autostereoscopic ?
Does the application require interaction with haptic stimuli ? Displays Technologies - Features 3D Displays and Optical Systems - Projection Systems
Considerations with regard to stereo image projection CRT LCD TFT M M D Lightwalv Addressability 4 kPixel 2 kPixel 2 kPixel 1.3 kPixel 4 kPixel Contrast h ig h lo w h ig h h ig h h ig h Time-multiplexing with active shutters: Colors very good m edium good very good very good Dimensions huge sm all/m edium sm all/m edium sm all m edium • both front and retro projection possible Refresh <180 Hz 60 Hz 60 Hz 60 (180Hz) 140 Hz Costs low average high high very high • active glasses are quite expensive (if many are required) • very high speed projector is required (light valve technology, expensive)
Polarized filtered images: • projection screen must preserve polarization (aluminized silver screen) • retro projection not yet possible (no suitable screen material available) • glasses are very cheap • two projectors are required (can cause image alignment problems)
FULLY IMMERSIVE SPHERICAL PROJECTION SYSTEM (THE CYBERSPHERE) RealityVision´s autostereoscopic display http://www.ndirect.co.uk/~vr-systems/sphere1.htm using holographic optical elements
Contact: David Trayner or Edwina Orr.RealityVision Ltd. 6 Yorkton St. London E2 8NH, UK: +44 (0)171 7399700 F: +44 (0)171 739 9707 E: [email protected]