A RealTime PhotoRealistic

Visual Flythrough

1;2 1 1 1

Daniel CohenOr Eran Rich Uri Lerner and Victor Shenkar

1

Tiltan System Engineering

BneiBrak Israel

2

Scho ol of Mathematical Sciences

TelAviv University RamatAviv Israel

Abstract

In this pap er we present a comprehensive ythrough system which

generates photorealistic images in true realtime The high p erfor

mance is due to an innovative rendering algorithm based on a discrete

ray casting approach accelerated by ray coherence and multiresolu

tion traversal The terrain as well as the D ob jects are represented by

a textured mapp ed based mo del The system is based on a pure

software algorithm and is thus p ortable It was rst implemented on a

workstation and then p orted to a generalpurp ose parallel architecture

to achieve realtime p erformance

Keywords Terrain Visualization Parallel Rendering Flight Simulator Vi

sual Simulations VoxelBased Mo deling Ray Casting

Intro duction

The quest for realtime photorealistic rendering has b een one of the ma jor

goals of D computer graphics in recent years Techniques for adding real

ism to the image such as shading shadow textures and transparency have

b een develop ed The generation of realistic images in realtime is currently

b eing researched Flight simulator applications have always led the way

in realtime p erformance Sp ecialpurp ose machines dedicated to ight

simulation have b een develop ed These machines generate images with rea

sonable realism in realtime but are exp ensive more than a few million US

dollars The main contribution of the work presented in this pap er is

that the realtime p erformance was achieved on commercial generalpurp ose

parallel architecture as opp osed to sp ecialized rendering hardware

Generating images of arbitrary complex scenes is not within the reach of

current technology However the rate of image generation in ight simulation

can achieve realtime b ecause the scenes that are viewed from the sky are not

to o complex Typical views contain terrains which are merely D or D

ob jects which are seen as relatively simple featureless ob jects Nevertheless

simulating photorealistic aerial views in realtime is by no means easy

The term visual ythrough can b e distinguished from ight simulation

Visual ythrough generates simulated images as seen from a video camera

attached to a ying ob ject The camera generates photorealistic images

although not necessarily in color since many video cameras have a greylevel

output It should b e emphasized that the generation of true photorealistic

images is critical for applications where the user needs to recognize the area

or identify ob jects on the ground ie targeting mission rehearsal See the

photorealistic impression of the images presented in Figure

In a typical ythrough scenario the camera views a very large area es

p ecially when the camera pitch angle is high ie towards the horizon In

many applications the camera ies at high sp eed over long distances and the

area covered during a few seconds of ight is vast This suggests that it is

not p ossible to load the entire terrain data onto the main memory For some

applications even a Gigabyte of RAM is not enough It is safe to say that

no size will ever suce since the application demands will always increase

according to the availability of space This suggests that ythroughs require

a large secondary storage together with a fast paging mechanism

An image of an aerial view gains its realistic impression by mapping a

digital photograph onto the terrain mo del In order to achieve high quality

full resolution of b oth the digital terrain mo del and the corresp onding aerial

photograph need to b e employed This causes a ma jor load on conventional

graphics hardware based on a geometric pip eline First a high resolution

p olygonal terrain mo del contains a vast numb er of p olygons which need to b e

pro cessed by the geometric pip eline while pro cessing tiny p olygons loses the

costeectiveness of the rasterization hardware The high resolution photo

graph that needs to b e texturemapp ed during rasterization creates a further

problem since large photographic maps need to b e loaded onto an exp ensive

cache For example the RealityEngine b oard cannot hold more than a few

Kilobytes of texture while larger textures need to b e paged from the

main memory in realtime To avoid that many ight simulators use

rep etitive patterns as ground texture but for some applications where a sp e

cic target area is a vital requirement a true photograph has to b e mapp ed

on the terrain These photographs are huge and must b e loaded on the y

to the rasterization hardware forming a serious b ottleneck in the rendering

pip eline

Instead of using a p olygonal mo del and a geometric pip eline we have

favored a software solution where the mo del is represented by a voxelbased

representation The texturemapping of the photograph over the mo del is a

prepro cessing stage which is decoupled from the rendering stage Voxel

based mo deling also lends itself to representing ne grained geometry The

voxel data is regular internally represented in an array and oers easy and

fast access to the data Each voxel represents and stores some discrete

lo cal attributes of the mo del representing the terrain contain a height

value and a color value while the voxels representing a D mo del contain

a texture photograph as will b e describ ed b elow in Section A voxel

based visual ight simulator with realtime p erformance has b een develop ed

at Hughes Training Inc Their ight simulator runs on sp ecial purp ose

hardware but yields p o or results on a graphics workstation

The visual ythrough that we have develop ed is hardware indep endent

and thus p ortable Portability is imp ortant since it enables integration of

the ythrough system with rapid progress in hardware platforms However

a software rendering algorithm must b e fast enough around a second or two

p er frame running on a sequential machine so that on a parallel machine

with pro cessors it achieves a rate higher than frames p er second That

Figure Two aerial photorealistic images generated by the ythough

is of course assuming little overhead is imp osed on the parallel version of

the algorithm

Although we have employed a parallel machine the real time p erformance

is mainly due to an innovative rendering algorithm The new algorithm gen

erates a photorealistic image such as in Figure within two seconds on a

common workstation The implementation of a parallel version of the al

gorithm on a way multipro cessor architecture has sp ed up the rendering

to achieve the desired realtime rates It should b e noted that other hard

ware indep endent ray casting algorithms have reached reasonable sp eeds

but just for p oint sampling Avoiding aliasing artifacts is quite

involved and time costly The algorithm presented here resembles the prin

ciples of the pro jection algorithm in However their algorithm is based

on a forward mapping metho d and was designed to b e implemented in hard

ware The algorithm presented in the next section is a simple ray casting

Figure The image footprint over the terrain is dened by the viewing

parameters

forward mapping accelerated by ray coherence and multiresolution traver

sal and highly optimized for hardware indep endent implementation

The remainder of this pap er is structured as follows Section describ es

the rendering algorithm Section presents the IBM Power Visualization

System the current parallel platform of the parallel algorithm implementa

tion details concerning the parallelization of the algorithm and some results

We discuss the generation of voxelbased ob jects in Section and conclude

with a brief discussion on our current activity and some nal remarks

The Rendering Algorithm

The sequence of images generated by the rendering algorithm is indep endent

Each image is dened by the lo cation of the camera in D space the camera

eld of view and its orientation namely the pitch roll yaw angles and image

resolution Figure depicts the image footprint dened by the pro jection

of the image frame over the terrain The terrain mo del is represented by

a voxelbased map generated from a discrete elevation map colored by a

corresp onding aerial photo map The rendering algorithm is based on a

backward mapping approach known as ray casting The image is generated

by casting a rayofsight emanating from the viewp oint through each of the

image pixels towards the mo del see Figure The ray traverses ab ove the

Figure Discrete ray casting of a voxelbased terrain

terrain voxels until it intersects the terrain The terrain color is sampled

and mapp ed back to the source pixel Since the mo del is discrete there is no

explicit intersection calculation but a sequential search for a hit b etween

the ray and a voxel The sp eed of the ray traversal is crucial for achieving

realtime p erformance

The technique we employ is based on a discrete grid traversal where the

steps along the ray are p erformed on the pro jection of the ray on the plane

rather than in D The heights along the ray are incrementally and uniformly

sampled and compared to the height of the terrain b elow it until a hit o ccurs

and the color of the terrain at the hit p oint is mapp ed back to the source

pixel If there is no hit then the background color of the sky is mapp ed

This apparently naive traversal is at in contrast to a hierarchical

traversal In a pyramidal elevation map is used The multiresolution

pyramid is treated as a hierarchy of b ounding b oxes through which the ray

traverses in a recursive topdown traversal The numb er of steps of the

hierarchical traversal is prop ortional to the height of the pyramid When

a binary pyramid is used the numb er of steps is logarithmic to the terrain

length rather than linear to the terrain size as in the case of the at traversal

Figure Assuming the terrain has no caves each ray can emanate from the

previous hit point

Our algorithm is based on the incremental at traversal but as will b e

shown some rays are hierarchically traversed

Since the terrain is a height eld map we can assume that the terrain

mo del has no vertical cavities or overhangs ie a vertical line has only

one intersection with the terrain The traversal can b e accelerated using ray

coherence The basic idea is that as long as the camera do es not roll

a ray cast from a pixel vertically adjacent always hits the terrain at a greater

distance from the viewp oint than that of the ray b elow it The image pixels

are generated column by column from b ottom to top A ray i emanating

ab ove ray i will always traverse a distance not shorter than the distance of

ray i see Figure Thus ray i can start its traversal from a distance

equal to the range of the previous hit of ray i This feature shortens the rays

traversal considerably

The total numb er of steps required to generate one column is equal to

the length of the column fo otprint eliminating the factor of the numb er of

pixel columns In other words a naive generation of one column has a time

complexity of O ml where l is the length of the column fo otprint and m

is the numb er of pixels in the image column Using ray coherence the time

complexity is reduced to O l only providing an order of magnitude sp eed

up The rays emanating from the b ottom of the column cannot gain from a

previous hit and are thus accelerated by a hierarchical traversal

Using the ab ove vertical ray coherence b etween consecutive rays each

terrain voxel is virtually traversed once The time complexity of the traversal

is prop ortional to the numb er of voxels in the image fo otprint This is still a

huge numb er since the image fo otprint can extend to the horizon Moreover

this numb er is viewdep endent and causes instability of the frame generation

rate

Due to p ersp ective pro jection the rays diverge with the distance caus

Figure Multiresolution traversal The voxel map resolution corresponds to

the sampling rate

ing a nonuniform sampling rate of the terrain voxels by the rays The rays

emanating from the b ottom of the image frame hit the terrain at a closer

range than the upp er rays Assuming that the terrain dataset is represented

in a single resolution then close voxels tend to b e oversampled while far

voxels are undersampled Using a hierarchy of data resolutions improves the

sampling since the rays can adaptively traverse and sample voxels of an ap

propriate size prop ortional to the pixel fo otprint see Figure Optimally

in every step one pixel is generated In multiresolution traversal the voxel

sampling rate b ecomes prop ortional to the numb er of rays ie pixels and

the numb er of steps b ecomes indep endent of the viewing direction That

is the numb er of steps over the terrain is in the order of the image space

rather than the ob ject space Thus the adaptive hierarchical traversal not

only sp eeds up the rendering but also helps to stabilize the frame generation

rate

In our implementation we use a pyramid of data resolutions where each

level has half of the resolution of the level b elow it Using more resolutions

can b e even more successful in the sense of uniformity of the sampling

but then it would use more space Another advantage of a binary pyramid

is the simplicity of alternating b etween consecutive levels where the step

sizes are either multiplied or divided by two taking advantage of the integer

arithmetic of the traversal Moreover the pyramid oers a fast rst hit for

the rst rays which emanate from the b ottom row of the pixel array Those

rays cannot b enet from coherency with the previous rays For those rays a

topdown traversal of the hierarchy sp eeds up their rst hit

One imp ortant issue that must b e taken care of in a realtime hierarchical

rendering is creating a soft transition when switching b etween levels A sharp

transition is very noticeable and causes an aliasing eect of a wave that

sweeps over the terrain A simple solution is to interp olate b etween adjacent

hierarchies where the interp olation weights are dened by the distance

from the viewp oint to the sampled voxels Since the range gradually changes

so do the weights causing a soft transition

Synthetic ob jects such as trees buildings and vehicles can b e placed

over the terrain The D ob jects are represented by sticks a run of voxels

of three typ es uniform sticks which are colored by a single color like a

terrain voxel textured sticks which contain a vertical sequence of colored

voxels and complex sticks which are textured sticks but contain some semi

transparent or fully transparent voxels see Synthetic ob jects are then

describ ed by a set of adjacent sticks A ray which hits a textured stick

climbs onto the stick and maps back the stick texture to the screen When

a semitransparent value is encountered a secondary ray continues through

the voxel The results of the secondary ray are then blended with the values

of the primary ray according to the value of the semitransparent voxels In

many cases the transparency value indicates a cavity in the stick in this case

no blending is p erformed and the colors of the secondary rays are directly

mapp ed to the pixels

Since cavities cause the spawning of secondary rays it is clear that they

slow down the rendering pro cess One way to reduce cavities is to ll them

up at coarse resolutions assuming the cavities are small enough and their

contribution to the nal image is insignicant One should note that in

typical scenes only a small fraction of the sticks need to b e complex For

example when viewing wo o ds only the trees at the b oundary needs to b e

fully represented with their non convex parts while most of the other trees

are hidden and only their tops can b e seen

A typical scene contains many replicated ob jects placed at dierent lo ca

tions and orientations Thus many sticks are common to many ob jects A

complex voxel contains a p ointer instead of a color which p oints into a stick

table Each stick consists of a header and a sequence of values The header

contains several attributes like the stick typ e and the stick length

The Basic Algorithm

In this section we present in detail the basic algorithm that generates a

single column of the image The algorithm is based on a fast traversal of the

column fo otprint over the terrain The voxels along the fo otprint are tested

for visibility and the colors of the visible ones are sampled and mapp ed back

to the image column The pseudoco de is shown in Figure

Let E b e the lo cation of the eye and P the lo cation of a column pixel

The parametric equation of a ray emanating from E and passing through P

is v P tP E Denote the ray direction P E by Q Qx Qy Qz

Then for a given x the co ordinates along the ray are explicitly given by

z P z x P xQz Qx

and

y P y x P xQy Qx

Assuming the ray is X ma jor ie Qx Qy then the sequence of

the voxel co ordinates x y along Q is generated by a forward dierences

evaluation of the line equation

z z Qz Qx

i+1 i

y y Qy Qx

i+1 i

where x x S I GN Qx

i+1 i

Using xed p oint arithmetic the integral co ordinate of y denoted by by c

is retrieved by a shift op eration on the binary representation y while the

fraction part w y by c is used for linear interp olation at the sampling

p oint see b elow The hit b etween the ray Q and the terrain is detected by

j

comparing heig htx by c the height of the terrain ab ove x by c against z

If z heig htx by c then x y and z are incrementally up dated otherwise a

hit has b een detected The terrain color at x by c is sampled and mapp ed

to the pixel P and the pro cess pro ceeds to the next ray Q emanating

j j +1

from P

j +1

Since the terrain is a height eld the ray Q do es not hit the terrain

j +1

b efore it reaches the hit p oint of Q The algorithm continues to evaluate the

j

sequence of the x y co ordinates and their heights need to b e compared to

the height of ray Q see Figure The slop e Qz Qx and the height

j +1 image plane

Q i+1

Q i

Figure Climbing from the hit point of ray Q to ray Q

i i+1

of ray Q ab ove x is evaluated by Equation Note that a small error is

j +1

intro duced since the plane dened by the rays emanating from a column of

the image plane is not p erp endicular to the main plane and may b e slightly

slanted due to the p ersp ective pro jection However when the eld of view is

small say under degrees the error is insignicant

Let E b e the lo cation of the eye

Let P b e the lo cation of the b ottom pixel of the column

Let U p b e the vector direction of the image columns

Let Q P E b e the direction of the ray emanating from P

Assume Qx Qy and E is ab ove the terrain and x E x and y E y

Let n b e the distance b etween x and the end of the terrain

while n f while not reaching end of terrain

while z heig htx by cf test for a hit

w y by c yield the subvoxel weight

Color Samplex by c w sample the voxels

Pixelj Color back map the results

if column done return

P U p move up to next pixel

Q P E climb to the new ray

z P z x P x Qz Qx

g

Move on to the next voxel along the ray

x SIGNQx move along the ma jor axes

y QyQx incrementally up date the Y co ordinate

z QzQx incrementally up date the ray height

g

if n the sky is seen

color the rest of the pixels with the sky color

Figure The integer base incremental traversal

The function Samplex by c w samples the terrain colors at the integer

co ordinates of x However the resolution of the xed p oint values is higher

than that of the voxel space and the fraction value denoted by w yields the

subvoxel lo cation of the hit p oint The exact hit p oint lies on the vertical grid

line b etween x by c and x by c see Figure Thus the voxel colors

of x by c and x by c are linearly interp olated at w Since the size of the w

double step size

Figure The samples are always on the vertical grid lines where w indicates

the subvoxel vertical sample location The switch to a double step size must

occur at an even step

pixel fo otprint is ab out the size of a voxel this simple lter is satisfactory

The traversal algorithm has to switch to a lower resolution at some p oint

Since the steps are of unit size along the ma jor direction it is rather simple

to double the step size and resp ectively the ray vector and its slop es To

preserve the prop erty that the steps are always at integer co ordinates of the

ma jor axes the switching to a double step size at the lower resolution must

o ccur at an even step of the current resolution see Figure

The switch to a lower resolution o ccurs at the distance where the voxel

fo otprint in the image is narrower than a pixel In other words we avoid

undersampling the voxels Since the vertical eld of view is not equal to the

horizontal eld of view we consider only the horizontal one allowing vertical

oversampling or undersampling to o ccur in some rare cases In particular

when the viewing pitch angle is low ie shallow the pixel fo otprint tends to

elongate and may cause signicant undersampling Vertically sup ersampling

the pixels to comp ensate for elongated fo otprints is not to o costly since it

do es not require accessing a larger numb er of voxels We have implemented

a variation of sup ersampling where each pixel has b een sup ersampled by

parallel rays The relaxed assumption that the rays cast from a single pixel

are parallel enables ecient implementation without any signicant loss of

quality

Parallel Implementation

Sequential implementation of the rendering algorithm cannot deliver the de

sired realtime rates on contemp orary workstations It is vital to use a p ow

erful parallel machine not only to sp eed up the rendering but also to supp ort

the pro cessing of very large databases The application requires ying over

thousands of square kilometers including many D ob jects Taking into ac

count the hierarchical data structures the total amount of data is over

Gigabytes see b elow Moreover the relevant data ie the image fo otprint

must b e continuously loaded into main memory Thus the machine needs

to have very large rst and secondary memories and high sp eed channels

b etween them All these requirements need the supp ort of a machine with

high sp eed and very large storage capacity with large bandwidth busses

However a p ostpro cessor is used to further accelerate the image generation

rate and to enhance the image quality describ ed b elow

A blo ck diagram of the system is illustrated in Figure The IBM Power

Visualization System PVS is the parallel machine describ ed b elow It is

controlled by an IBM RS Supp ort Pro cessor which also serves as a con

nection to the external world It reads the commands from the users control

stick and sends control command from the PVS to a Post Rendering Pro ces

sor PRP see b elow through an Ethernet LAN The images generated by

the PVS are sent via an HIPPI MBSec channel to the PRP and are

displayed on a standard NTSC monitor

The IBM Power Visualization System

The IBM Power Visualization System PVS was designed to provide com

putational p ower highsp eed memory and IO to realize very large amounts

of complex data The PVS is a shared memory architecture consisting of up

to parallel pro cessing units and up to GB of internal lo cal and global

memory

The architecture consists of up to eight pro cessor cards Each pro cessor

card consists of four pro cessor elements each comp osed of an Intel iXR

or iXP micropro cessor op erating at or MHz

Pro cessor storage consists of MBytes of lo cal memory p er pro cessor

and global memory which can b e increased to MBytes The global

memory consists of up to four memory cards Each card is designed to PVS

Local i860 Memory

System Interface Bus Interface

System Bus 1280MB/sec

SCSI HIPPI Shared Memory Interface Interface 512MB/Card (up to 2GB)

100MB/sec Disk Array Post Rendering 4 SCSI F/W Drives Processor

Support Processor Ethernet Monitor RS/6000

Control Stick

Figure A block diagram of the system

provide a data bandwidth of MBsec or MBsec This is accomplished

by partitioning the memory into four interleaved memory banks each of

which can p erform memory reads and writes thus reducing the latency and

improving throughput In addition there is interleaving b etween cards if

there are multiple memory cards in the system

An SCSI interface card with four FastWide p eak of MBSec con

trollers is used to connect to the disk array Using an SCSI disk reduces the

system price and promises upgradability The PVS strips the data into all

the controllers giving a throughput of more than MBSec Thus it can

contain the database and can load the memory fast enough

The PVS also provides means for pro ducing and outputting the frames in

realtime A video controller which is attached via an HIPPI channel to the

Server includes two logically distinct frame buers with a total capacity of

up to MB The rst bit buer is used for workstation graphics and text

from an XWindows system The other is a bitpixel doublebuered full

color RGB image buer at HDTV resolutions and ab ove

Implementation Details

The rendering task is partitioned among the pro cessors One of the them

selected arbitrarily op erates as the master and the rest are the slaves The

master pro cessor among its many tasks sets the viewing parameters of the

next frame including the new p ositioning of the camera and its orientation

according to the tra jectories of the ight The generated image is treated

as a p o ol of columns and each slave pro cessor renders one column of pixels

as an atomic task As so on as a slave terminates rendering one column it

picks a new column from the p o ol Access to the p o ol is monitored by a

semaphore op eration provided by the PVS library The semaphore forces

exclusive access to the p o ol so that only one pro cessor at a time can pick

a column Moreover as so on as the last columns of the frames have b een

picked and generated the free pro cessors start to generate the rst columns

of the next frame Using this strategy the pro cessors are kept busy with a

p erfect load balancing

Although the PVS contains as much as two Gigabytes of RAM the

database cannot b e loaded entirely into the main memory The entire database

is stored in the disk array while the relevant sections ie the image fo otprint

are loaded dynamically into memory The terrain database is partitioned into

small square tiles According to the viewing parameters the master draws

the rectangular frame of the image fo otprint on the terrain makes sure that

the tiles that fall in the frame fo otprint are already in memory and loads the

missing tiles from the diskarray Since the fo otprint changes incrementally

only a few tiles need to b e loaded at each frame A large conguration of

the main memory consists of two Gigabytes and can contain more than the

size of one frame fo otprint thus we use an extended fo otprint Some of the

tiles that are in the larger fo otprint would otherwise have b een loaded on the

next frame Thus the extended fo otprint saves many critical loadings The

tiles that are loaded are actually prefetched and their presence is not critical

for correct rendering of the current frame This mechanism is found to b e

very ecient as it can treat fast changes of camera as much as one entire

eld of view p er second

Pitch p p p p p

Table Frames per second fps generated by the PVS as a function of the

number of processors Each line of the table shows the fps sampled at dierent

pitch angles

Results

Quantitative results are presented in Table The frame generation rate of

the PVS has b een measured at three dierent angles for dierent numb ers

of pro cessors These rates are further accelerated by the PRP to achieve a

steady frame rate of frames p er second However from these numb ers

we can learn ab out the p erformance of the algorithm First a linear sp eed

up is achieved The ab ove numb ers imply that by doubling the numb er of

pro cessors the frame generation rate is more than doubled This is b ecause

one pro cessor is dedicated as the master pro cessor A second observation is

the dep endency b etween the p erformance and the pitch angle As the pitch

angle gets smaller the frame generation rate decreases This is b ecause at

small pitch angles the frame fo otprint extends However since we use a

hierarchy the size of the fo otprint is b ounded It should b e noted that there

is a sp eed quality tradeo By scaling the pixeltovoxel ratio it is p ossible

to sp eed up the frame generation rate As the voxels used are scaled

the fo otprint sizes voxelwise decrease Of course as the pixeltovoxel ratio

increases the voxels are oversampled and the image is blurred However this

ratio is used as a to ol to tune the quality of the image as the frame generation

rate is guaranteed by the PRP

A typical database consists of a large terrain with tens of target areas

The global terrain is a meter resolution playground of x square kilo

meters which is Giga voxels Each voxel is four bytes thus the size of the

global terrain is Gigabytes Adding the hierarchy requires a third more

G thus G bytes in total Each target area consists of three levels of

detail x square kilometers of meter resolution by square

kilometers of meter resolution and by square meters of

centimeters for the highest resolution A single target area database size is

G bytes No hierarchy is needed b ecause the coarser levels are given in

the global terrain Assuming for example target areas require over G

bytes In total G bytes are needed for the terrain data The D ob jects

consume more space A typical ob ject requires ab out M bytes Here we

should mention that if true colors were needed and not only grey levels the

database would have b een almost double the size

The Post Rendering Pro cessor

The images generated by the PVS are asynchronous since their rate is dep en

dent on the viewing direction The frames are created at a rate of Hz

From these images an NTSC video signal should b e pro duced The image

elds that is the eveno dd NTSC rows have to b e transmitted at a rate of

Hz interlaced If the elds contain only the last frame generated by the

PVS the human eye would detect jumps every time the frame is changed

To achieve a smo oth sequence of images that do not irritate the eye it is

necessary to generate the frames at a synchronous rate The idea is to sim

ulate small changes in the camera p osition as D transformations applied to

the last frame available However unlike the interp olation metho d here

the image needs to b e extrap olated The image is digitally warp ed on the

y with resp ect to the ying tra jectories The warping is done using the

Datacube MaxVideo machine The MaxVideo serves as the Post Rendering

Pro cessor and is also used for some other D functions such as automatic

gain control AGC ltering scaling and rolling the image It should b e em

phasized that interp olating b etween available frames is not p ossible since it

would cause a small but critical latency which is not acceptable in realtime

systems where realtime feedback is vital The extrap olated images may

have minor dierences from the next real frame However since the ying

tra jectories are known and are relatively smo oth the transition b etween the

extrap olated frame to the real frame is smo oth Since the warping function

might mapp ed back a p oint outside the source frame the real frames are

slightly larger and include margins These margins are relatively small since

ying tra jectories are smo oth recalling that the real images are created at a

rate of Hz

Given an image A generated by some camera p osition the goal is to

warp the image so that it approximates the image B that would have b een

generated by a new camera p osition Let us dene f as the function that

maps B back to A such that if p is a p oint in the D space that is seen from

0 0

pixelx in B and in pixel x in A then f x x Once f is known the pixel

0

color at x is determined by bilinear interp olation at x

A p ersp ective warp function would b e b est however the MaxVideo sup

p orts a second degree p olynomial warp Thus f is comp osed of two functions

f f f where f and f are two second degree p olynomials

x y x y

2 2

f x y a a x a y a xy a x a y

x 1 2 3 4 5 6

and

2 2

f x y b b x b y b xy b x b y

y 1 2 3 4 5 6

To determine the ab ove co ecients a set of n equations is

explicitly dened by n control p oints The system of n equations is solved

using a least squares metho d The n equations are dened by calculating

the p osition of n p oints in the D world co ordinate for camera p osition A

and B and pro jecting them back to the image space We used nine p oints

evenly distributed in the image plane During rendering the D co ordinates

of the terrain p oint seen from those nine xed lo cations are registered

Denote the vector of unknown co ecients by C a b j

j j j

2 2

y The system that we need to solve is FC X where F x y x y x

i i i i i

i i

0 0

i n These are two sets of n equations for six and X x y

i i

i

variables Assuming n is larger than six the least squares solution gives us

1

t t 1

C F F F X Note that for n C F X

Note also that since the roll rotation is a simple D transformation it

can b e implemented directly using the MaxVideo warp er

Mo deling VoxelBased Ob jects

The pro cess known as voxelization converts a continuous geometry into a dis

crete representation Many existing mo dels are represented by a p olygon

mesh that approximates the real ob ject However for a photorealistic appli

cation photo mapping is essential see Figure This requires warping

the photograph of the ob ject so that it matches the D mo del and then ap

plying it as a texture map to the voxels Alternatively a sculpting technique

Figure Voxelbased objects houses trees and a tank

can b e employed Given a set of images of an ob ject from known directions

one can craft the shap e of the mo del by p eeling away the background voxels

around the pro jected images We start from a solid b ox of black voxels

Then given an image rays are cast from the background pixels back into the

voxels clearing the voxels encountered into background color Rep eating

this pro cess from many images which view the mo del from dierent direc

tions leaves the nonbackground voxels with the shap e of the mo del This

pro cess of reconstruction from pro jection yields the inher

ently by pro jecting the nonbackground pixels back towards the voxels by

means of ray casting

A simplied implementation of the ab ove sculpting technique has b een

employed We use only three photographs of a toy ob ject For example

the three photographs of a toy Scud are shown in Figure a These three

photographs are scaled down to the voxel space resolution as can b e seen in

Figure b At this stage the ob ject is separated from the background pixel

If this is not achieved by color thresholding a contour is drawn manually

around the ob ject The result of the sculpting pro cess and the photomapping

from these images is a D voxelbased textured ob ject which can b e rendered

from arbitrary viewing direction The images shown in Figures and are

rendered very close to the ob ject in order to observe the ne details Note

a b

Figure A toy Scud a Three photographs sidefronttop b the images

after scaling to the voxel space resolution

Figure The voxelized Scud from three dierent viewing directions

Figure Three photographs of a T tank

Figure The voxelized T from three dierent viewing directions

that the resolution of the ob ject is higher than of the terrain However these

ob jects are to b e seen from a distance as shown in the previous images

Current Porting Activity

The development pro ject was started in while the PVS was stateof

theart but since then the pro cessing p ower of a single pro cessor has grown

by a factor of compared to the i

Although the p erformance achieved on the PVS is satisfactory it is clear

that a faster platform will allow us to deal b etter with higher resolutions and

more and more ob jects of richer detail The p ortability of the application

p ermits the adoption of a new parallel shared memory architecture according

to the b ehavior of the commercial market

Using a distributed memory machine was ruled out since the application

was designed for shared memory architecture The only company that man

ufactures shared memory architecture in the same price range as the PVS

is Silicon Graphics Inc SGI SGIs machines have a similar architecture

to the PVS with the exception that SGI uses a large cache MBytes in

contrast to the MBytes of the PVSs lo cal memory

SGI oers the Challenge with a maximum of RMhz CPUs

and the Power Challenge with a maximum of RMhz CPUs The

Power Challenge was designed for oating p oint applications and each CPU

is more than twice as fast in such applications In integer applications the

R and R have the same p erformance giving the Challenge double

the p erformance of the Power Challenge Both machines can store up to

GBytes internally and up to TBytes externally

The primary results on the SGI Challenge indicate a sp eed up of ab out

times faster than the PVS while the scalability remains linear This is

achieved with only minor changes in the co de used for the PVS mainly to

comp ensate for the absence of lo cal memory

Final Remarks

We have presented a discrete ray casting algorithm accelerated by ray coher

ence and multiresolution traversal The time complexity of the algorithm is

prop ortional to the numb er of image pixels which can b e regarded as con

stant The combination of the ecient rendering algorithm and the p owerful

parallel machine results in a realtime photorealistic visual ythrough The

parallel rendering task partitions the image space among the PVS pro ces

sor elements putting the load at the scene space stored in the PVS shared

memory Due to data prefetching the wide bandwidth of the busses linear

sp eedup has b een observed as well as hardly any read or write contentions in

the shared memory We have achieved p erfect load balancing by overlapping

b etween frames

It should b e noted that the sequential version of the rendering algorithm

runs well under two seconds on an SGI workstation for a terrain size that can

t into main memory It is exp ected that in the future with the progress of

memory bandwidth and CPU sp eed visual ythrough will b e able to run in

realtime on advanced sequential workstations

Acknowledgments

This work was develop ed at Tiltan System Engineering in collab oration with

IBM Israel Our thanks to Meir NissimNir who built the terrain database

and to Sylvia Kohn who built the ob jects and develop ed some new ideas

during the pro cess We thank all the stu at IBM who help ed us along in

many dierent ways

References

K Akeley Reality Engine graphics In Proceedings of SIGGRAPH

pages ACM

ES Chen and L Williams View interp olation for image synthesis In

Proceedings of SIGGRAPH pages

D Cohen and C Gotsman Photorealistic terrain imaging and ight

simulation IEEE Computer Graphics and Applications

March

D Cohen and A Kaufman D scanconversion algorithms for linear

and quadratic ob jects In A Kaufman editor Volume Visualization

pages IEEE Computer So ciety Press Los Alamitos CA

D Cohen and A Shaked Photorealistic imaging of digital terrains

Computer Graphics Forum

S Co quillart and M Gangnet Shaded display of digital maps IEEE

Computer Graphics and Applications July

Evans and Sutherland Computer Corp oration Esig technical

overview Technical rep ort Komas Drive Salt Lake CityUT

J Folby M Zyda D Pratt and R Mackey Npsnet Hierarchical data

structures for realtime three dimensional visual simulation Computers

and Graphics

A Kaufman D Cohen and R Yagel Volume graphics IEEE Com

puter July

YG Leclerc and SQ Lau Terravision A terrain visualization system

Technical rep ort Technical Rep ort SRI international April

C Lee and YG Shin An ecient metho d for terrain ren

dering Pacic Graphics pages

FK Musgrave Grid tracing Fast ray tracing for height elds Technical

rep ort Department of Mathematics Yale University Decemb er

DW Paglieroni and SM Petersen Height distributional distance trans

form metho ds for height eld ray tracing ACM Transactions on Graph

ics Octob er

L Williams Pyramidal parametrics In Computer Graphics volume

pages

J Wright and J Hsieh A voxelbased forward pro jection algorithm

for rendering surface and volumetric data In AE Kaufman and GM

Nielson editors Proceedings of Visualization pages IEEE

Computer So ciety Press

Bio

Daniel CohenOr is senior lecturer at the Department of Computer Science

in TelAviv University His research interests include rendering techniques

volume visualization architectures and algorithms for voxelbased graphics

He received a BSc Cum Laude in b oth Mathematics and Computer Science

an MSc Cum Laude in Computer Science from BenGurion

University and a Phd from the Department of Computer Science at

State University of New York at Stony Bro ok

Dr CohenOr has extensive industrial exp erience in he designed a

realtime ythrough at Tiltan System Engineering during he worked

on the development of a new parallel architecture at Terra Computer and

recently he has b een working with MedSim Ltd on the development of an

ultrasound simulator He can b e reached at the Department of Computer Sci

ence TelAviv University Israel His email address is danielmathtauacil

Eran Rich received the BSc degree in Electrical Engineering from the

TelAviv University in Since then he has b een working at Tiltan Sys

tem Engineering leading the development group of the ythrough and other

applications His main areas of interest are parallel computing computer

graphics and image pro cessing He is a memb er of IEEE Computer So ciety

and IEEE

Uri N Lerner is a do ctoral candidate in Computer Science in Stanford

University starting Septemb er He received his BSc summa cum

laude in b oth Mathematics and Computer Science from TelAviv University

in He joined the Tiltan System Engineering in and has led the

ythrough team for the last year His current interests include computer

graphics and parallel computing with a sp ecial emphasis on voxelized terrain

rendering and antialiasing metho ds

Victor Shenkar is the General Manager of Tiltan System Engineering

Ltd in BneiBrak Israel He received a BSc and a MSc in Electrical

Engineering from the Technion Israel Institute of Technology and a Phd

from Stanford University His current interest include accelerated image

rendering and digital photogrammetry