A Theory of Catadioptric Image Formation

Shree K Nayar and Simon Baker

Technical Rep ort CUCS

Department of Computer Science

Columbia University

New York NY

Email fnayarsimonbgcscolumbiaedu

Abstract

Conventional video cameras have limited elds of view whichmake them restrictive

for certain applications in computational vision A catadioptric sensor uses acombination

of lenses and mirrors placed in a carefully arranged conguration to capture a muchwider

eld of view When designing a catadioptric sensor the shap e of the mirrors should ideally

b e selected to ensure that the complete catadioptric system has a single eective viewp oint

The reason a single viewp oint is so desirable is that it is a requirement for the generation

of pure p ersp ective images from the sensed images In this pap er we derive and analyze

the complete class of singlelens singlemirror catadioptric sensors which satisfy the xed

viewp oint constraint Some of the solutions turn out to be degenerate with no practical

value while other solutions lead to realizable sensors We also derive an expression for the

spatial resolution of a catadioptric sensor and include a preliminary analysis of the defo cus

blur caused by the use of a curved mirror

Keywords Image formation sensor design sensor resolution defo cus blur

Intro duction

Many applications in computational vision require that a large eld of view is imaged

Examples include surveillance teleconferencing and mo del acquisition for virtual reality

Moreover a number of other applications such as egomotion estimation and tracking

could b enet from enhanced elds of view Unfortunately conventional imaging systems

are severely limited in their elds of view and so b oth researchers and practitioners have

had to resort to using either multiple or rotating cameras in order to image the entire scene

A recently develop ed and eectiveway to enhance the eld of view is to use mirrors

in conjunction with lenses This approach to image capture is fast gaining in p opularity

see for example Nayar Yagi and Kawato Hong Goshtasby and

Gruver Yamazawa et al Nalwa Nayar and Baker and Nayar

We refer to the general approach of using mirrors in combination with conventional

imaging systems as catadioptric image formation Recent work in this area has led to the

development of a truly omnidirectional video camera with a spherical eld of view Nayar

Yamazawa et al Nalwa and Nayar it is highly As noted in

desirable that a catadioptric system or in fact any imaging system have a single view

p oint center of pro jection The reason a single viewp oint is so desirable is that it p ermits

the generation of geometrically correct p ersp ective images from the images sensed by

the catadioptric cameras This is p ossible b ecause under the single viewp oint constraint

every pixel in the sensed images measures the irradiance of the light passing through the

viewp oint in one particular direction Since weknow the geometry of the catadioptric sys

Dioptrics is the science of refracting elements lenses whereas catoptrics is the optics of reecting

surfaces mirrors The combination of refracting and reecting elements is therefore referred to as cata

dioptrics Hecht and Za jac

tem we can precompute this direction for each pixel Therefore we can map the irradiance

value measured by each pixel onto a plane at any distance from the viewp oint to form a

p ersp ective image This p ersp ective image can subsequently be pro cessed using the vast

arrayoftechniques develop ed in the eld of computational vision which assume p ersp ective

pro jection Moreover if the image is to be presented to a human as in Peri and Nayar

it needs to b e a p ersp ective image in order to not app ear distorted Naturallywhen

the catadioptric imaging system is omnidirectional in its eld of view a single viewp oint

p ermits the construction of panoramic images in addition to p ersp ective ones

In this pap er we b egin in Section by deriving the entire class of catadioptric

systems with a single eective viewp oint and which are constructed just using a single

conventional lens and a single mirror As we will show the parameter family of mirrors

which can be used is exactly the class of rotated swept conic sections Within this class

of solutions several swept conics prove to be degenerate solutions and hence impractical

while others lead to practical sensors During our analysis we will stop at many p oints

to evaluate the merits of the solutions as well as the merits of closely related catadioptric

sensors prop osed in the literature

An imp ortant prop erty of a sensor that images a large eld of view is its resolution

The resolution of a catadioptric sensor is not in general the same as that of the sensors used

to construct it In Section we study why this is the case and derive an expression for the

relationship between the resolution of a conventional imaging system and the resolution

of a derived catadioptric sensor This expression should be carefully considered when

constructing a catadioptric imaging system in order to ensure that the nal sensor has

sucient resolution It could also b e used to design conventional sensors with nonuniform

resolution which when used in an appropriate catadioptric system have a sp ecied eg

uniform resolution

Another optical prop erty which is mo died by the use of a catadioptric system is

fo cusing It is a well known fact that a curved mirror increases image blur Hecht and

Za jac and so in Section we analyze the eect which the use of a curved mirror has

on defo cus blur Two factors combine to cause blur in catadioptric systems the nite

size of the lens and the curvature of the mirror We rst analyze how the interaction of

these two factors causes defo cus blur and then present some preliminary numerical results

for one sp ecic mirror shap e the hyp erb oloid The results show that the fo cal setting of a

catadioptric sensor using a curved mirror may be substantially dierent to that needed in

aconv entional sensor Moreover our analysis illustrates a numerical metho d of computing

the relationship b etween the two fo cus settings which could b e used to precompute a fo cus

lo okup table for a sp ecic catadioptric sensor

The Fixed Viewp oint Constraint

The xed viewp oint constraint is a requirement that a catadioptric sensor only measure

the intensity of light passing through a single point in D space The direction of the

light passing through this p ointmay vary but that is all In other words the catadioptric

sensor can only sample the D plenoptic function Adelson and Bergen Gortler et

al at a single p oint in D space The xed D p ointatwhich a catadioptric sensor

samples the plenoptic function is known as the eective viewpoint

Supp ose we use a single conventional camera as the only sensing element and a

single mirror as the only reecting surface If the camera is an ideal p ersp ective camera

and we ignore defo cus blur it can b e mo deled bythepoint through which the p ersp ective

pro jection is p erformed ie the eective pinhole Then the xed viewp oint constraint

requires that each ray of light passing through the eective pinhole of the camera which

was reected by the mirror would have passed through the eective viewp oint if it had

not b een reected by the mirror

Derivation of the Fixed Viewp oint Constraint Equation

Without loss of generalitywe can assume that the eective viewp oint v of the catadioptric

system lies at the origin of a cartesian co ordinate system Supp ose that the eective pinhole

is lo cated at the p oint p Then also without loss of generalitywe can assume that the z axis

z lies in the direction vp Moreover since p ersp ective pro jection is rotationally symmetric

ab out any line through p the mirror can be assumed to be a surface of revolution ab out

the z axis z Therefore wework in the D cartesian frame v r z where r is a unit vector

orthogonal to z and try to nd the dimensional prole of the mirror z r z x y where

p

r x y Finallyifthe distance from v to p is denoted by the parameter c we have

v and p c See Figure for an illustration of the co ordinate frame

We b egin the translation of the xed viewp oint constraintinto symb ols by denoting

the angle between an incoming ray from a world point and the r axis by Supp ose that

this ray intersects the mirror at the point z r Then since it also passes through the

origin v w e have the relationship

z

tan

r



In Figure we have drawn the image plane as though it were orthogonal to the z axis z indicating

that the optical axis of the camera is anti parallel to z In fact the eective viewp oint v and the axis of

symmetry of the mirror prole z r need not necessarily lie on the optical axis Since p ersp ective pro jection

is rotationally symmetric with resp ect to anyray that passes through the pinhole p the camera could b e

rotated ab out p so that the optical axis is not parallel to the z axis and moreover the image plane can b e

rotated indep endently so that it is no longer orthogonal to z In this second case the image plane would

be nonfrontal This do es not pose any additional problem since the mapping from a nonfrontal image

plane to a frontal image plane is onetoone image of world point image plane

effective pinhole, p=(0,c)

world point c normal

γ β γ+β α θ z

zˆ mirror point, (r,z)

effective viewpoint, v=(0,0) rˆ r = x 2 + y 2

Figure The geometry used to derive the xed viewp oint constraint equation The viewp oint

v is lo cated at the origin of a D co ordinate frame v r z and the pinhole of the camera

p c is lo cated at a distance c from v along the z axis z If a rayoflight whichwas ab out

to pass through v is reected at the mirror p ointrz the angle b etween the ray of lightandr

z

is tan If the ray is then reected and passes through the pinhole p the angle it makes

r

cz dz

with r is tan and the angle it makes with z is Finallyif tan

r dr

is the angle b etween the normal to the mirror at rz and z then by the fact that the angle of

incidence equals the angle of reection wehave the constraint that

If we denote by the angle between the reected ray and the negative r axis we also

have

c z

tan

r

since the reected ray must pass through the pinhole p c Next if is the angle

between the z axis and the normal to the mirror at the pointrz we have

dz

tan

dr

Our nal geometric relationship is due to the fact that we can assume the mirror to be

sp ecular This means that the angle of incidence must equal the angle of reection So

o

if is the angle between the reected ray and the z axis we have and

o

See Figure for an illustration of this constraint Eliminating

from these two expressions and rearranging gives

Then taking the tangent of both sides and using the standard rules for expanding the

tangent of a sum

tan A tan B

tanA B

tan A tan B

we have

tan tan tan

tan tan tan

Substituting from Equations and yields the xed viewpoint constraint equation

dz

c z r

dr

dz

r cz z

dr

which when rearranged is seen to b e a quadratic rstorder ordinary dierential equation

dz dz

r z c r cz z r c z

dr dr

General Solution of the Constraint Equation

The rst step in the solution of the xed viewp oint constraint equation is to solve it as a

quadratic to yield an expression for the surface slop e

q

r c z r cz z r cz

dz

dr r z c

c c

The next step is to substitute y z and set b which yields

q

y r b r b y r b

dy

dr ry

Then we substitute rx y r b which when dierentiated gives

dy dx

y x r r

dr dr

and so we have

p

rx r dx r b r x

r x r

dr r

Rearranging this equation yields

dx

p

dr r

b x

Integrating b oth sides with resp ect to r results in

p

ln r C b x ln x

where C is the constant of integration Hence

p

k

x b x r

C

where k e is a constant By back substituting rearranging and simplifying

we arrive at the two equations which comprise the general solution of the xed viewp oint

constraint equation

c c k k

z r k

k

c k c c

k r z

k

In the rst of these two equations the constant parameter k is constrained by k rather

than k since k leads to complex solutions

Sp ecic Solutions of the Constraint Equation

Together the two relations in Equations and represent the entire class of mirrors

that satisfy the xed viewp oint constraint A quick glance at the form of these equations

reveals that the mirror proles are all conic sections Hence the D mirrors themselves are

swept conic sections However as we shall see although every conic section is theoretically

a solution of one of the two equations a number of them prove to be impractical and

only some lead to realizable sensors We will now describ e each of the solutions in detail

Since the class of solutions forms a parameter c and k familywe consider them in the

following order

Planar Solutions Equation with k and c

Conical Solutions Equation with k and c

Spherical Solutions Equation with kand c

Ellipsoidal Solutions Equation with k and c

Hyp erb oloidal Solutions Equation with kand c

There is one conic section which we have not mentioned the parab ola Although

the parab ola is not a solution of either Equation or Equation for nite values of

c

c and k it is a solution of Equation in the limit that c k and h a

k

constant Under these limiting conditions Equation tends to

h r

z

h

As shown in Nayar this limiting case corresp onds to orthographic pro jection More

over in that setting the parab ola do es yield a practical omnidirectional sensor with a num

ber of advantageous prop erties In this pap er we restrict attention to the p ersp ective case

and refer the reader to Nayar for a discussion of the orthographic case However

most of the results can either be extended to or applied directly to the orthographic case

Planar Mirrors

In Solution if we set k and c we get the crosssection of a planar mirror

c

z

As shown in Figure this plane is the one which bisects the line segment vp joining the

viewp oint and the pinhole

The converse of this result is that for a xed viewp oint v and pinhole p there is

only one planar solution of the xed viewp oint constraint equation The unique solution is

the p erp endicular bisector of the line joining the pinhole to the viewp oint

p v

x p v

To prove this it is sucient to consider a xed pinhole p a planar mirror with unit

normal n and a point q on the mirror Then the fact that the plane is a solution of the

xed viewp oint constraint implies that there is a single eective viewp oint v v n q

Moreover the eective viewp oint is the reection of the pinhole p in the mirror ie the

single eective viewp oint is

v n q p p q n n

Since the reection of a single point in two dierent planes is always two dierent p oints

the p erp endicular bisector is the unique planar solution

An immediate corollary of this result is that for a single xed pinhole no two dierent

planar mirrors can share the same viewp oint Unfortunately a single planar mirror do es image of world point image plane

effective pinhole, p=(0,c)

world point

c 2

mirror, z = c/2

c 2

zˆ

effective viewpoint, v=(0,0) rˆ

c

Figure The plane z is a solution of the xed viewp oint constraint equation Conversely

it is p ossible to show that given a xed viewp oint and pinhole the only planar solution is the

p erp endicular bisector of the line joining the pinhole to the viewp oint Hence for a xed pinhole

two dierent planar mirrors cannot share the same eective viewp oint For each such plane the

eective viewp ointis the reection of the pinhole in the plane This means that it is imp ossible

to enhance the eld of view using a single p ersp ective camera and an arbitrary number of planar

mirrors while still resp ecting the xed viewp oint constraint If multiple cameras are used then

solutions using multiple planar mirrors are p ossible Nalwa

not enhance the eld of view since discounting o cclusions the same camera moved from

p to v and reected in the mirror would have exactly the same eld of view It follows

that it is imp ossible to increase the eld of view by packing an arbitrary number of planar

mirrors p ointing in dierent directions in front of a conventional imaging system while

still resp ecting the xed viewp oint constraint On the other hand in applications such

a as stereo where multiple viewp oints are a necessary requirement the multiple views of

scene can b e captured by a single camera using multiple planar mirrors See for example

Goshtasby and Gruver

This brings us to the panoramic camera prop osed byNalwa Nalwa To ensure

a single viewp oint while using multiple planar mirrors Nalwa Nalwa has arrived at

a design that uses four separate imaging systems Four planar mirrors are arranged in a

squarebased pyramid and each of the four cameras is placed ab ove oneofthefaces of the

pyramid The eective pinholes of the cameras are moved until the four eective viewp oints

ie the reections of the pinholes in the mirrors coincide The result is a sensor that has

a single eective viewp oint and a panoramic eld of view of approximately The

panoramic image is of relatively high resolution since it is generated from the four images

captured by the four cameras Nalwas sensor is straightforward to implement but requires

four of each comp onent ie four cameras four lenses and four digitizers It is p ossible

to use only one digitizer but at a reduced frame rate

Conical Mirrors

In Solution if we set c and k we get a conical mirror with circular cross

section

s

k

z r

See Figure for an illustration of this solution The angle at the ap ex of the cone is

where

s

tan

k

This might seem like a reasonable solution but since c the pinhole of the camera must

b e at the ap ex of the cone This implies that the only rays of lightentering the pinhole from

the mirror are the ones which graze the cone and so do not originate from nite extent

ob jects in the world see Figure Hence the cone with the pinhole at the vertex is a

degenerate solution of no practical value

The cone has b een used for wideangle imaging Yagi and Kawato Yagi and

Yachida In these implementations the pinhole is placed quite some distance from

the ap ex of the cone It is easy to show that in such cases the viewp oint is no longer a

single p oint Nalwa If the pinhole lies on the axis of the cone at a distance e from

the ap ex of the cone the lo cus of the eective viewp oint is a circle The radius of the circle

is easily seen to b e

e cos

If the circular lo cus lies inside b elow the cone if the circular lo cus lies

outside ab ove the cone and if the circular lo cus lies on the cone

Spherical Mirrors

we get the spherical mirror In Solution if we set c and k

k

z r

Like the cone this is a solution with little practical value Since the viewp oint and pinhole

coincide at the center of the sphere the observer sees itself and nothing else as is illustrated

in Figure imaged world point image plane

zˆ effective pinhole, p=(0,0) effective viewpoint, v=(0,0)

rˆ τ τ

mirror

world point

Figure The conical mirror is a solution of the xed viewp oint constraint equation Since the

pinhole is lo cated at the ap ex of the cone this is a degenerate solution of little practical value If

the pinhole is moved away from the ap ex of the cone along the axis of the cone the viewp oint

is no longer a single p oint but rather lies on a circular lo cus If is the angle at the ap ex of the

cone the radius of the circular lo cus of the viewp ointis e  cos where e is the distance of the

pinhole from the ap ex along the axis of the cone Further if the circular lo cus lies inside

b elow the cone if the circular lo cus lies outside ab ove the cone and if the

circular lo cus lies on the cone

pinhole, p viewpoint, v

Figure The spherical mirror satises the xed viewp oint constraint when the pinhole lies at

the center of the sphere Since c the viewp oint also lies at the center of the sphere Like

the conical mirror the sphere is of little practical value b ecause the observer can only see itself

rays of light emitted from the center of the sphere are reected back at the surface of the sphere

directly towards the center of the sphere

The sphere has b een used to enhance the eld of view for landmark navigation

Hong In this implementation the pinhole is placed outside the sphere and so there

is no single eective viewp oint The lo cus of the eective viewp ointcan be computed in a

a quite complex straightforward manner using a symbolic mathematics package but it is

function of the distance between the center of the sphere and the pinhole The lo cus is of

comparable size to the mirror Spheres have also been used in stereo applications Nayar

but as describ ed b efore multiple viewp oints are a requirement for stereo and hence

the xed viewp oint constraintisnot critical

Ellipsoidal Mirrors

In Solution when kand c we get the ellipsoidal mirror

c

z r

a b

e e

where

s

s

k c k

a and b

e e

The ellipsoid is the rst solution that can actually be used to enhance the eld of view

As shown in Figure if the viewp oint and pinhole are at the fo ci of the ellipsoid and the

mirror is taken to b e the section of the ellipsoid that lies b elowthe viewp oint ie z

the eective eld of view is the entire upp er hemisphere z The ellipsoid has never

b efore been prop osed as away of enhancing the eld of view of a video camera

Hyp erb oloidal Mirrors

In Solution when kand c we get the hyp erb oloidal mirror

c

r z

a b

h h

where

s

s

c c k

and b a

h h

k k

As seen in Figure the hyp erb oloid also yields a realizable solution The curvature of

the mirror and the eld of view both increase with k In the other direction in the limit

k the hyp erb oloid attens out to the planar mirror of Section Yamazawa et al

Yamazawa et al Yamazawa et al recognized that the hyp erb oloid is indeed

a practical solution They have implemented a sensor designed for autonomous navigation

and demonstrated the generation of p ersp ective images image of world point image plane

p=(0,c)

world point c

zˆ

rˆ

v=(0,0)

Figure The ellipsoidal mirror satises the xed viewp oint constraint when the pinhole and

viewp oint are lo cated at the two fo ci of the ellipsoid If the ellipsoid is terminated by the horizontal

plane passing through the viewp oint z the eld of view is the entire upp er hemisphere z

It is also p ossible to cut the ellipsoid with other planes passing through v but it app ears there

is to b e little to b e gained by doing so The ellipsoid has never b efore b een prop osed as a wayof

enhancing the eld of view of a video camera image of world point image plane

effective pinhole, p=(0,c)

world point

c

zˆ

v=(0,0) rˆ

k=8.0

Figure The hyp erb oloidal mirror satises the xed viewp oint constraint when the pinhole

and the viewp oint are lo cated at the two fo ci of the hyp erb oloid This solution do es pro duce

the desired increase in eld of view The curvature of the mirror and hence the eld of view

increase with k In the limit k  the hyp erb oloid attens to the planar mirror of Section

Yamazawa et al Yamazawa et al Yamazawa et al recognized that the hyp erb oloid

is indeed a practical solution and have implemented a sensor designed for autonomous navigation

Resolution of a Catadioptric Sensor

In this section we assume that the conventional camera used in the catadioptric sensor

has a frontal image plane lo cated at a distance u from the pinhole and that the optical

axis of the camera is aligned with the axis of symmetry of the mirror See Figure for

an illustration of this scenario Then the denition of resolution which we will use is the

following Consider an innitesimal area dA on the image plane If this innitesimal pixel

images an innitesimal solid angle d of the world the resolution of the sensor is

dA

d

dA

Note that the resolution will vary as a function of the p oint on the image plane at the

d

center of innitesimal area dA

If is the angle made between the optical axis and the line joining the pinhole to

the center of the innitesimal area dA see Figure the solid angle subtended by the

innitesimal area dA at the pinhole is

dA cos dA cos

d

u cos u

Therefore the resolution of the conventional camera is

dA u

d cos

Then the area of the mirror imaged by the innitesimal area dA is

d c z dA c z cos

dS

cos cos u cos

where is the angle between the normal to the mirror at rz and the line joining the

pinhole to the mirror p ointrz Since reection at the mirror is sp ecular the solid angle

of the world imaged by the catadioptric camera is

dS cos dA c z cos

d

r z u r z pixel area, dA image of world point image plane

optical axis

u solid angle, dω ψ

focal plane

pinhole, p=(0,c) normal

world point solid angle, dω φ φ

c mirror area, dS

mirror point, (r,z) solid angle, dυ zˆ

rˆ

viewpoint, v=(0,0)

Figure The geometry used to derive the spatial resolution of a catadioptric sensor Assuming

the conventional sensor has a frontal image plane which is lo cated at a distance u from the pinhole

and the optical axis is aligned with the z axis z the spatial resolution of the conventional sensor

dA u

is Therefore the area of the mirror imaged by the innitesimal image plane area dA



d

cos

cz cos

 dA So the solid angle of the world imaged by the innitesimal area dA on is dS

u cos

cz cos

the image plane is d  dA Hence the spatial resolution of the catadioptric sensor

u r z

u r z cz

r z dA dA

since cos  is

d d

cz cos r cz cz r

Hence the resolution of the catadioptric camera is

dA dA u r z r z cos

d c z cos c z d

But since

c z

cos

c z r

we have

dA dA r z

d c z r d

Hence the resolution of the catadioptric camera is the resolution of the conventional camera

used to construct it multiplied by a factor of

r z

c z r

where rz is the p oint on the mirror b eing imaged

c

The rst thing to note from Equation is that for the planar mirror z the

resolution of the catadioptric sensor is the same as that of the conventional sensor used to

construct it This is as exp ected by symmetry Naturally the resolution in b oth cases do es

as seen in Equation Finally Equations vary across the image plane like



cos

and can in principle be used to design image detectors with nonuniform resolution

which comp ensate for the ab ove variation and pro duce a catadioptric sensor with uniform

resolution across the eld of view More generally any sp ecied variation in resolution

could be achieved

Defo cus Blur of a Catadioptric Sensor

Two factors combine to cause defo cus blur in catadioptric imaging systems the nite

size of the lens and the curvature of the mirror In this section we investigate this

eect for the hyp erb oloid mirror Generalization to the other mirrors is straightforward

We pro ceed by considering a xed p oint in the world and a xed p ointinthelens We nd

the point on the curved mirror which reects a ray of light from the world point through

the lens p oint Then we compute where on the image plane this mirror p oint is imaged

By considering the lo cus of imaged mirror p oints as the lens pointvaries we can compute

the area of the image plane onto which axed world p ointisimaged

the D cartesian frame To p erform this analysis we need to work in D We use

v x y z where v is the lo cation of the eective viewp oint p is the lo cation of the eective

pinhole z is a unit vector in the direction vp and the vectors x and y are orthogonal unit

vectors in the plane z As b efore we assume that the eective pinhole is lo cated at

a distance c from the eective viewp oint Moreover as in Section we assume that the

conventional camera used in the catadioptric sensor has a frontal image plane lo cated at

a distance u from the pinhole and that the optical axis of the camera is aligned with the

z axis Finallywe assume that the eective pinhole of the lens is lo cated at the center of

the lens and that the lens has a circular ap erture See Figure for an illustration of this

conguration

l

Consider a p oint m x y z on the mirror and a point w x y z in the

kmk

world where l kmk Then since the hyp erb oloid mirror satises the xed viewp oint

constraint a ray of light from w which is reected by the mirror at m passes directly

through the center of the lens ie the pinhole This rayoflight is known as the principal

ray Hecht and Za jac Next supp ose a ray of light from the world p oint w is

on the mirror and then passes through the lens reected at the point m x y z

p oint l d cos d sin c In general this ray of light will not b e imaged at the same

p oint on the image plane as the principal ray When this happ ens there is defo cus blur

The lo cus of the intersection of the incoming rays through l and the image plane as l varies

over the lens is known as the blur region or region of confusion Hecht and Za jac blur region image plane, z=c+u

pinhole, p=(0,0,c) u lens focal plane, z=c

l=(d⋅cosλ,d⋅sinλ,c) l world point, w= m (x,y,z)

principal ray

c v normal, n

m 1 =(x 1 ,y 1 ,z 1 )

mirror m=(x,y,z)

zˆ focused plane, z=c-v

plane, z=0

viewpoint, v=(0,0,0)

Figure The geometry used to analyze the defo cus blur We work in the D cartesian frame

v x y z where x and y are orthogonal unit vectors in the plane z In addition to the

assumptions of Section we also assume that the eective pinhole is lo cated at the center of

the lens and that the lens has a circular ap erture If a ray of light from the world p oint w

l

x y z is reected at the mirror p oint m x y z and then passes through the lens

kmk

point l d  cos d  sin c there are three constraints on m it must lie on the mirror

the angle of incidence must equal the angle of reection and the normal n to the mirror

at m and the twovectors l m and w m must b e coplanar

For an ideal thin lens the blur region is circular and so is often referred to as the blur circle

Hecht and Za jac As well shall see for a catadioptric sensor using a curved mirror

the shap e of the blur region is not in general circular

If we know the p oints m and lwe can nd the p oint on the image plane where the

ray of light through these p oints is imaged First the line through m in the direction lm

is extended to intersect the focused plane By the thin lens law Hecht and Za jac

the fo cused plane is the plane

f u

z c v c

u f

where f is the fo cal length of the lens and u is the distance from the fo cal plane to the image

plane Since all p oints on the fo cused plane are p erfectly fo cused the p ointofintersection

on the fo cused plane can be mapp ed onto the image plane using p ersp ective pro jection

Hence the x and y co ordinates of the intersection of the ray through l and the image plane

are the x and y co ordinates of

u v

m l l

v c z

and the z co ordinate is the z co ordinate of the image plane c u

l

Given the lens point l d cos d sin cand the world p oint w x y z

kmk

there are three constraints on the p oint m x y z First m must lie on the mirror

and so we have

k c k c

x y z

k

Secondly the incidentrayw m the reected raym l and the normal to the mirror

at m must lie in the same plane The normal to the mirror at m lies in the direction

n k x k y c z

Hence the second constraint is

n w m l m

Finally the angle of incidence must equal the angle of reection and so the third constraint

on the p oint m is

n w m n l m

kw m k kl m k

These three constraints on m are all multivariate p olynomials in x y and z Equa

tion and Equation are b oth of order and Equation is of order We were

unable to nd a closed form solution to these three equations Equation has terms

in general and so it is probable that none exists but we did investigate numerical solutions

The results are presented in Figures and

For the numerical solutions we set c meter used the hyp erb oloid mirror with

k and assumed the radius of the lens to be centimeters We considered the p oint

m on the mirror and set the distance from the viewp oint to the world

p oint w to b e l meters In Figure we plot the area of the blur region on the ordinate

against the distance to the fo cused plane v on the abscissa The smaller the area of the

blur region the b etter fo cused the image will be We see from Figure that the area

never reaches exactly and so an image formed using this catadioptric sensor can never

b e p erfectly fo cused The same is true whenever the mirror is curved However it should

be emphasized that the minimum area is very small and in practice there is no problem

fo cusing the image for a single point

to note that the distance at which the image of the world p oint It is interesting

will be best fo cused ie somewhere in the range meters is much less than the

distance from the pinhole to the world point approximately meter from the pinhole to

the mirror and then meters from the mirror to the world point The reason for this is

that the mirror is convex and so tends to increase the divergence of rays coming from the

world p oint

Finallywe provide an explanation of the fact that there are two minima of the blur 0.0002 Hyperboloid Mirror 0.00018

0.00016

0.00014

0.00012

0.0001

8e-05

6e-05

4e-05 Area of Blur Region at Unit Distance (meters squared)

2e-05

0 1 1.05 1.1 1.15 1.2 1.25 f ⋅ u Distance to the Focused Plane v = (in meters)

(u - f)

Figure The area of the blur region plotted against the distance to the fo cused plane v

fracf  uu f for a p oint m on the hyp erb oloid mirror with k In this

example we have c meter the radius of the lens centimeters and the distance from the

viewp ointto the world point l meters The area never b ecomes exactly and so the image

can never b e p erfectly fo cused However the area do es become very small and so fo cusing on a

single p oint is not a problem in practice There are two minima in the area which corresp ond to

the two dierent foldings of the blur region illustrated in Figures Also note that the distance

at which the image will be b est fo cused meters is much less than the distance from

the pinhole to the world p oint approximately meter from the pinhole to the mirror and then

meters from the mirror to the world p oint The reason is that the mirror is convex and so tends

to increase the divergence of rays of light coming from the world p oint w

area in Figure As mentioned b efore for an isolated conventional lens the blur region is a

circle In this case as the fo cus setting is adjusted to fo cus the lens all p oints on the blur

circle move towards the center of the blur circle at a rate which is prop ortional to their

distance from the center of the blur circle Hence the blur circle steadily shrinks until the

lens is p erfectly fo cused If the fo cus setting is moved further in the same direction the

blur circle grows again as all the p oints on it move away from the center On the other

hand in the catadioptric case the sp eed with which points move is dep endent on their

p osition in a more complex way Moreover the direction in which the p oints are moving is

not a constant This b ehavior is illustrated in Figures af

In Figure a the fo cused plane is meters from the pinhole and the image

is quite defo cused As the fo cused plane moves to meters away in Figure b the

p oints in the left half of Figure a are moving upwards more rapidly than those in the

right half the p oints in the right half are also moving upwards Further the p oints in the

left half are moving upwards more rapidly than they are moving towards the right This

eect continues in Figures c and d In Figure d all the p oints are still moving

horizontally towards the center of the blur region but vertically they are nowmoving away

from the center The points continue to move horizontally towards the center of the blur

region but with those in the left half again moving faster This causes the blur region

to overlap itself as seen in Figure e Finally for the fo cused plane at meters

in Figure f all p oints are moving away from the center of the blur region in b oth

directions The blur region is expanding and the image b ecoming more defo cused

Using the numerical techniques of this section it would be simple to compute a

lo okup table for converting the fo cus settings of the conventional camera into ones for the

catadioptric sensor The most imp ortant prop erty of such a table would be how quickly

may be dicult to fo cus a large it changes across the mirror If it changes to o quickly it

region of the catadioptric image at one time without using sophisticated comp ensating 0.0008 0.0003

0.0006 0.0002

0.0004

0.0001 0.0002

0 0

-0.0002 -0.0001

-0.0004

-0.0002 -0.0006

-0.0008 -0.0003

-0.003 -0.002 -0.001 0 0.001 0.002 0.003 0.004 -0.003 -0.002 -0.001 0 0.001 0.002 0.003 0.004

a meters b meters

0.00015 0.0004

0.0003 0.0001

0.0002

5e-05 0.0001

0 0

-0.0001 -5e-05

-0.0002

-0.0001 -0.0003

-0.00015 -0.0004

-0.002 -0.001 0 0.001 0.002 0.003 0.004 -0.002 -0.001 0 0.001 0.002 0.003

c meters d meters

0.004 0.005

0.004 0.003

0.003 0.002 0.002

0.001 0.001

0 0

-0.001 -0.001

-0.002 -0.002 -0.003

-0.003 -0.004

-0.004 -0.005

0 0.0002 0.0004 0.0006 0.0008 0.001 -0.0002 0 0.0002 0.0004 0.0006 0.0008 0.001 0.0012 0.0014

e meters f meters

Figure The variation in the shap e of the blur region as the distance to the fo cused plane is

varied All of the blur regions in this gure are relatively well fo cused Note that the scale of the

gures are all dierent and that the scale on the two axes is dierent in general

lenses Intuitively the more curved the mirror the more quickly the conversion table will

vary However investigating to what extent this is the case is left for a future study

Summary

In this pap er we have studied three design criteria of catadioptric sensors the shap e

of the mirrors the resolution of the cameras and the fo cus settings of the cameras

In particular wehave derived the complete class of mirrors that can b e used with a single

camera found an expression for the resolution of a catadioptric sensor in terms of the

resolution of the conventional camera used to construct it and presented a preliminary

analysis of defo cus blur

References

Adelson and Bergen EH Adelson and JR Bergen The plenoptic function and

elements of early vision In Landy and Movshon editors Computational Models of

Visual Processing chapter MIT Press

Gortler et al SJ Gortler R Grzeszczuk R Szeliski and M Cohen The lumi

graph In Computer Graphics Proceedings Annual Conference Series pages ACM

SIGGRAPH

Goshtasby and Gruver A Goshtasby and WA Gruver Design of a singlelens

cognition stereo camera system Pattern Re

Hecht and Za jac E Hecht and A Za jac Optics AddisonWesley

Hong J Hong Image based homing In Proceedings of the IEEE International

Conference on Robotics and Automation May

Nalwa VS Nalwa A true omnidirectional viewer Technical rep ort Bell Lab ora

tories Holmdel NJ USA February

Nayar and Baker Shree K Nayar and Simon Baker Catadioptric image formation

In Proceedings of the DARPA Image Understanding WorkshopNew Orleans May

Nayar SK Nayar Sphereo Recovering depth using a single camera and t wo sp ecu

lar spheres In Proceedings of SPIE Optics Il lumination and Image Sensing for Machine

Vision IINovember

Nayar SK Nayar Omnidirectional video camera In Proceedings of the

DARPA Image Understanding Workshop May A related pap er will app ear in

CVPR

Peri and Nayar V Peri and SK Nayar Generation of p ersp ective and panoramic

video from omnidirectional video In Proceedings of the DARPA Image Under

May standing Workshop New Orleans

Yagi and Kawato Y Yagi and S Kawato Panoramic scene analysis with conic

pro jection In Proceedings of the International ConferenceonRobots and Systems

Yagi and Yachida Y Yagi and M Yachida Realtime generation of environmental

map and obstacle avoidance using omnidirectional image sensor with conic mirror In

Proceedings of the Conference on Computer Vision and Pattern Recognition pages

June

Yamazawa et al K Yamazawa Y Yagi and M Yachida Omnidirectional imaging

with hyp erb oloidal pro jection In Proceedings of the International ConferenceonRobots

and Systems

Yamazawa et al K Yamazawa Y Yagi and M Yachida Obstacle avoidance with

omnidirectional image sensor Hyp erOmni Vision In Proceedings of the IEEE Interna

tional Conference on Robotics and Automation pages May