Intro duction

The general problem of achieving corresp ondence or optical ow as it is known in the motion

literature is to recover the D displacement eld b etween p oints across two images Typical

applications for which full corresp ondence that is corresp ondence for all image p oints is

initially required include the measurement of motion stereopsis structure from motion D

reconstruction from p oint corresp ondences and recently visual recognition active vision and

computer graphics animation

In this pap er we fo cus on two problemsone theoretical and the other more practical

On the practical side we address the problem of establishing the full p ointwise displacement

eld b etween two views greylevel images of a general D ob ject We achieve this by rst

considering a theoretical problem of establishing a quadric surface reference frame on the

ob ject In other words given any two views of some unknown textured opaque quadric

3

surface in D pro jective space P is there a nite numb er of corresp onding p oints across

the two views that uniquely determine all other corresp ondences coming from p oints on the

quadric A constructive answer to this question readily suggests that we can asso ciate a

virtual quadric surface with any D ob ject not necessarily itself a quadric and use it for

describing shap e but more imp ortantly for achieving full corresp ondence b etween the two

views

On the conceptual level we prop ose combining geometric constraints captured from

knowledge of a small numb er of corresp onding p oints manually given for example and pho

tometric constraints captured by the instantaneous spatiotemp oral changes in image light

intensities conventional optical ow The geometric constraints we prop ose are related

to the virtual quadric surface mentioned ab ove These constraints lead to a transformation

a nominal quadratic transformation that is applied to one of the views with the result of

bringing b oth views closer together The remaining displacements residuals are recovered

by using the spatial and temp oral derivatives of image light intensityeither by correlation

of image patches or by optical ow techniques

The Quadric Reference Surface

3

We consider ob ject space to b e the D pro jective space P and image space to b e the D

2

pro jective space P b oth over the eld C of complex numb ers Views are denoted by

i

0

indexed by i The epip oles are denoted by v and v and we assume their lo cation

1 2

denotes equality up to a scale is known for metho ds see for example The symb ol

GL stands for the group of n n matrices P GL is the group dened up to a scale and

n n

S P GL is the symmetric sp ecialization of P GL

n n

2 3

Result Given two arbitrary views P of a quadric surface Q P C with

1 2

0 3 0

centers of pro jection at O O P and O O Q then ve corresp onding p oints across

the two views uniquely determine all other corresp ondences

0 0 0

Proof Let x x x and x x x b e co ordinates of and resp ectively and z

0 1 2 1 2 0

0 1 2

z b e co ordinates of Q Let O then the quadric surface may b e given as

3

the lo cus z z z z and as the pro jection from O onto the plane

0 3 1 2 1

z In case where the centers of pro jection are on Q the line through O meets Q in

3

exactly one other p oint and thus the mapping Q is generically onetoone and so

1

2

has a rational inverse x x x x x x x x x x Because all quadric surfaces of

0 1 2 0 1 0 2 1 2

0

the same rank are pro jectively equivalent we can p erform a similar blowup from with

2

02 0 0 0 0 0 0

the result x x x x x x x The pro jective transformation A P GL b etween the two

4

0 0 1 0 2 1 2

representations of Q can then b e recovered from ve corresp onding p oints b etween the two

images

The result do es not hold when the centers of pro jection are not on the quadric surface

2

This is b ecause the mapping b etween Q and P is not onetoone a ray through the center

of pro jection meets Q in two p oints and therefore a rational inverse do es not exist We are

interested in establishing a more general result that applies when the centers of pro jection

are not on the quadric surface One way to enforce a onetoone mapping is by making

opacity assumptions dened b elow

Denition Opacity Constraint Given an ob ject Q fP P g we assume there

1 n

exists a plane through the camera center O that do es not intersect any of the chords P P

i j

ie Q is observed from only one side of the camera Secondly we assume that the surface

is opaque which means that among all the surface p oints along a ray from O the closest

p oint to O is the p oint that also pro jects to the second view The rst constraint

2

therefore is a camera opacity assumption and the second constraint is a surface opacity

assumption together we call them the opacity constraint

3

With an appropriate reparameterization of P we can obtain the following result

2 3

Theorem Given two arbitrary views P of an opaque quadric surface Q P

1 2

then nine corresponding points across the two views uniquely determine al l other correspon

dences

The following auxiliary prop ositions are used as part of the pro of

0 0 0 0

be four p p p Lemma Relative Ane Parameterization Let p p p p and p

o 1 2 3

3 2 1 o

corresponding points coming from four noncoplanar points in space Let A be a col lineation

0 0 2 0

v Final ly let v be scaled of P determined by the equations Ap p j and Av

j

j

0 0 3 0

such that p Ap v Then for any point P P projecting onto p and p we have

o

o

0 0

p Ap k v

The coecient k k p is independent of ie is invariant to the choice of the second

2

view and the coordinates of P are x y k

The lemma its pro of and its theoretical and practical implications are discussed in detail

in The scalar k is called a relative ane invariant and can b e computed with the aid of

a second arbitrary view For future reference let stand for the plane passing through

2

P P P in space

1 2 3

Pro of of Theorem From Lemma any p oint P can b e represented by the co ordinates

x y k and k can b e computed from Equation Since Q is a quadric surface then there

>

exists H S P GL such that P HP for all p oints P of the quadric Because H

4

is symmetric and determined up to a scale it contains only nine indep endent parameters

Therefore given nine corresp onding image p oints we can solve for H as a solution of a

0

linear system each corresp onding pair p p provides one linear equation in H of the form

>

x y k H x y k

Given that we have solved for H the mapping due to the quadric Q can

1 2

0

b e determined uniquely ie for every p we can nd the corresp onding p as

1 2

>

follows The equation P HP gives rise to a second order equation in k of the form

2

ak bpk cp where the co ecient a is constant dep ends only on H and the

co ecients b c dep end also on the lo cation of p Therefore we have two solutions for k

br

0 1 2

where and by Equation two solutions for p The two solutions for k are k k

2a

p

2

r b ac The nding shown in the next auxiliary lemma is that if the surface Q is

opaque then the sign of r is xed for all p Therefore the sign of r for p that leads to

1 o

a p ositive ro ot recall that k determines the sign of r for all other p

o 1

p

2

Lemma Given the opacity constraint the sign of the term r b ac is xed for al l

points p

1

Proof Let P b e a p oint on the quadric pro jecting onto p in the rst image and let the

1 2 1 2

OP intersect the quadric at p oints P P and let k k b e the ro ots of the quadratic ray

2

equation ak bpk cp The opacity assumption is that the intersection closer to O

0

is the p oint pro jecting onto p and p

0

Recall that P is a p oint on the quadric in this case used for setting the scale of v in

o

Equation ie k Therefore all p oints that are on the same side of as P have

o o

p ositive k asso ciated with them and vice versa There are two cases to b e considered either

P is b etween O and ie O P or is b etween O and P ie O P that

o o o o

1 2

is O and P are on opp osite sides of In the rst case if k k then the nonnegative ro ot

o

1 2

is closer to O ie k maxk k If b oth ro ots are negative the one closer to zero is closer

1 2

to O again k maxk k Finally if b oth ro ots are p ositive then the larger ro ot is closer

1 2

to O Similarly in the second case we have k mink k for all combinations Because

P can satisfy either of these two cases the opacity assumption gives rise to a consistency

o

requirement in picking the right ro ot either the maximum ro ot should b e uniformly chosen

for all p oints or the minimum ro ot

In the next section we will show that Theorem can b e used to surround an arbitrary

D surface by a virtual quadric ie to create quadric reference surfaces which in turn can

b e used to facilitate the corresp ondence problem b etween two views of a general ob ject The

remainder of this section takes Theorem further to quantify certain useful relationships

b etween the centers of two cameras and the family of quadrics that pass through arbitrary

congurations of eight p oints whose pro jections on the two views are known

3 2

Theorem Given a quadric surface Q P projected onto views P with centers

1 2

0 3

of projection O O P there exists a parameterization of the image planes that

1 2

yields a representation H S P GL of Q such that h when O Q and the sum of the

4 44

0

elements of H vanishes when O Q

Proof The reparameterization describ ed here was originally intro duced in as part of

3

the pro of of Lemma We rst assign the standard co ordinates in P to three p oints

0

on Q and to the two camera centers O and O as follows We assign the co ordinates

to P P P resp ectively and the co ordinates

1 2 3

0

0

OO to O O resp ectively By construction the p oint of intersection of the line

with has the co ordinates

0

Let P b e some p oint on Q pro jecting onto p p The line OP intersects at the p oint

The co ordinates can b e recovered up to a scale by the mapping

1

0

as follows Given the epip oles v and v we have by our choice of co ordinates that p p p

1 2 3

2

and v are pro jectively in P mapp ed onto e e e and

1 2 3

e resp ectively Therefore there exists a unique element A P GL that satises

1 3

0

e Denote A p Similarly the line A p e j and A v O P

1 1 j j 1

0 0 0 0

e j and intersects at Let A P GL b e dened by A p

j 2 3 2

j

0 0 0 0 0

e Further let A p A v

2 2

1

A It is easy to see that A A where A is the collineation dened in Lemma

1

2

Likewise the homogeneous co ordinates of P are transformed into k With this new

co ordinate representation the assumption O Q translates to the constraint that h

44

> 0

H and the assumption O Q translates to the constraint

>

H

Corollary Theorem provides a quantitative measure of proximity of a set of eight D

points projecting onto two views to a quadric that contains both centers of projection

Proof Given eight corresp onding p oints we can solve for H with the constraint

>

H This is p ossible since a unique quadric exists for any set of nine p oints in

0

general p osition the eight p oints and O The value of h is then indicative of how close

44

the quadric is from the other center of pro jection O

2

Note that when the camera center O is on the quadric then the leading term of ak

bpk cp vanishes a h and we are left with a linear function of k We see that

44

it is sucient to have a birational mapping b etween Q and only one of the views without

employing the opacity constraint This is b ecause of the asymmetry intro duced in our

metho d the parameters of Q are reconstructed with resp ect to the frame of reference of the

rst camera ie relative ane reconstruction in the sense of rather than reconstructed

pro jectively Also note the imp ortance of obtaining quantitative measures of proximity of an

eightp oint conguration of D p oints to a quadric that contains b oth centers of pro jection

this is a necessary condition for observing a critical surface A sucient condition is that

the quadric is a hyp erb oloid of one sheet Theorem provides therefore a to ol for

analyzing part of the question of how likely are typical imaging situations within a critical

volume

Achieving Full Corresp ondence b etween Views of a General

D Ob ject

In this section we derive an application of Theorem to the problem of achieving full cor

resp ondence b etween two greylevel images of a general D ob ject The basic idea similar

to is to treat the corresp ondence problem as comp osed of two parts a nominal

transformation with the aid of a small numb er of known corresp ondences and a residual dis

placement eld that is recovered using instantaneous spatiotemp oral derivatives of image

intensity values along epip olar lines This paradigm is general in the sense that it applies

to any D ob ject However it is useful if the nominal transformation brings the two views

closer together

Consider for example the case where the nominal transformation is a homography of

2

P of some plane In that case the residual eld is simply the relative ane invariant

k of Equation In other words if the ob ject is relatively at then the k eld is small

and thereby the nominal transformation which is Ap brings the two views closer to each

other If the ob ject is not at however then the residual eld may b e large within regions

in the image that corresp ond to ob ject p oints that are far away from This situation is

demonstrated in the second row display in Figure Three p oints were chosen two eyes

and the right mouth corner for the computation of the planar nominal transformation The

overlay of the second view and the transformed rst view demonstrate that the central region

of the face is brought closer at the exp ense of regions near the b oundary which corresp ond

to ob ject p oints that are far away from the virtual plane passing through b oth eyes and the

mouth corner

This example naturally suggests that a nominal transformation based on placing a virtual

quadric reference surface on the ob ject would give rise to a smaller residual eld note that the

planar transformation is simply a particular case of a quadric transformation The nominal

quadratic transformation can b e formalized as a corollary of Theorem as follows

Corollary of Theorem A virtual quadric surface can be tted through any D sur

face not necessarily a quadric surface by observing nine corresponding points across two

views of the object

Proof First it is known that there is a unique quadric surface through any nine p oints in gen

n (n+1)(n+2)21

eral p osition This follows from a Veronese map of degree two v P P

2

I I

dened by x x x where x ranges over all monomials of degree two in

0 n

3 9

x x For n this is a mapping from P to P taking hyp ersurfaces of degree two

0 n

3 9

in P ie quadric surfaces into hyp erplane sections of P Thus the subset of quadric

3 9

surfaces passing through a given p oint in P is a hyp erplane in P and since any nine hy

9

p erplanes in P must have a common intersection there exists a quadric surface through

any given nine p oints If the p oints are in general p osition this quadric is smo oth ie H is

of full rank

Therefore by selecting any nine corresp onding p oints across the two views we can apply

the construction describ ed in Theorem and represent the displacement b etween corresp ond

0

ing p oints p and p across the two views as follows

0 0 0

p Ap k v k v

q r

where k k k k is the relative ane structure of the virtual quadric and k is the

q r q r

remaining parallax which we call the residual The term within parentheses is the nominal

Figure Row Two views of a face on the left and on the right Row The

1 2

lefthand display shows the overlayed edges of the two views ab ove The righthand display

shows the overlay of the ane transformed and The three p oints chosen were the two

1 2

eyes and the right mouth corner Notice that the displacement across the center region of

the face was reduced at the exp ense of the p eripheral regions that were taken farther apart

Row The lefthand display shows followed by the nominal quadratic transformation

1

The righthand display shows the overlay of edges of the display on the left and Notice

2

that the original displacements b etween and are reduced to ab out pixels Row

1 2

The display on the left shows the overlay of the target view on the transformed view

2

nominal quadratic followed by residual ow The righthand display shows the eect

1

of a nominal quadric transformation due to a hyp erb oloid of two sheets This unintuitive

solution due to an unsuccessful choice of sample p oints creates the mirror image on the right

side

0

quadratic transformation and the remaining term k v is the unknown displacement along

r

the known direction of the epip olar line Therefore Equation is the result of representing

the relative ane structure of a D ob ject with resp ect to some reference quadric surface

namely k is a relative ane invariant b ecause k and k are b oth invariants by Theorem

r q

Note that the corollary is analogous to describing shap e with resp ect to a reference plane

instead of a plane we use a quadric and use the to ols describ ed in the previous

section in order to establish a quadric reference surface The overall algorithm for achieving

full corresp ondence given nine corresp onding p oints p p p is summarized b elow

o 1 8

0

Determine the epip oles v v using eight of the corresp onding p oints and recover the

0 0 0

collineation A from the equations Ap p j and Av v Scale v to satisfy

j

j

0 0

p Ap v

o

o

0 0

Compute k j from the equation p Ap k v

j j j

j

>

Compute the quadric parameters from the nine equations x y k H x y k

j j j j j j

k j Note that k k k

o 1 2 3

2

For every p oint p compute k as the appropriate ro ot of k of ak bpk cp where

q

>

the co ecients a b c follow from x y k H x y k and the appropriate

q q q q q q

2

bp k cp consistent with the ro ot ro ot follows from the sign of r for ak

o o o

o

k

o

0

Ap k v Warp according to the nominal transformation p

q 1

0

The remaining displacement residual b etween p and p consists of an unknown displace

ment k along the known epip olar line The spatiotemp oral derivatives of image light

r

intensity can b e used to recover k

r

In case and only then the ray OP do es not intersect the quadric the solutions for the

corresp onding k are complex numb ers ie p cannot b e repro jected onto p due to Q This

case can b e largely avoided when the nine sample p oints are spread as far apart as p ossible

on the view of the ob ject see for analytic results and computer simulations on the

1

existence and distribution of complex p o ckets

Also a tight t of a quadric surface onto the ob ject can b e obtained by using many

corresp onding p oints to obtain a least squares solution for H Note that from a practical

p oint of view we would like the quadric to lie as close as p ossible to the ob ject otherwise

the algorithm though correct would not b e useful ie the residuals may b e larger than the

original displacement eld b etween the two views

Exp erimental Results on Real Images

We have implemented the metho d describ ed in the previous section for purp oses of computer

simulation and for application in a real image situation The computer simulations are shown

in and some of the real image exp eriments are shown here

Figure top row shows two images of a face taken from two distinct viewp oints Achiev

ing full corresp ondence b etween two views of a face is extremely challenging for two reasons

First a face is a complex ob ject that is not easily parameterized Second the texture of

a typical face do es not contain enough image structure for obtaining p ointtop oint corre

sp ondence in a reliable manner There are a few p oints on the order of that can

b e reliably matched such as the corners of the eye mouth and eyebrows We rely on these

few p oints to p erform the quadratic nominal transformation and the epip olar geometry and

then apply optical ow techniques to nish o the corresp ondence everywhere else The

optical ow metho d we used was a mo dication on the technique describ ed in which

is a coarsetone gradientbased metho d following

The epip oles were recovered using the algorithm describ ed in using a varying numb er

of p oints The results presented here used the minimal numb er of nine p oints but similar

p erformance was obtained using more than nine p oints with a least squares solution for H

From Figure second row lefthand display we see that typical displacements b etween

corresp onding p oints around the center region of the face vary around pixels Figure

third row lefthand display shows the quadric nominal transformation applied to the view

The overlay of the edges of view and the transformed view are shown in the righthand

1 2

display One clearly sees that b oth the center of the face and the b oundaries are brought

closer together Typical displacements have b een reduced to around pixels The optical

ow algorithm restricted along epip olar lines was applied b etween the transformed view and

the second view The nal displacement eld nominal transformation due the quadric plus

the residuals recovered by optical ow was applied to the rst view to yield a synthetic image

that if successful should lo ok much like the second view In order to test the similarity

b etween the synthetic image and the second view the overlay of the edges of the two images

is shown in the fourth row of Figure lefthand display

Finally to illustrate that a quadric surface may yield unintuitive results we show in

the fourth row of Figure righthand display the result of having a hyp erb oloid of two

sheets as a quadric reference surface This is accidental but evidently can happ en with an

unsuccessful choice of sample p oints

Summary

Part of this pap er addressed the theoretical question of establishing a onetoone mapping

b etween two views of an unknown quadric surface We have shown that nine corresp onding

p oints are sucient to obtain a unique map provided we make the assumption that the

surface is opaque We have also shown that an appropriate parameterization of the image

planes facilitates certain questions of interest such as the likeliho o d that eight corresp onding

p oints will b e coming from a quadric lying in the vicinity of b oth centers of pro jection

On the practical side we have shown that the to ols develop ed for quadrics can b e applied

to any D ob ject by setting up a virtual quadric surface lying in the vicinity of the ob ject The

quadric serves as a reference frame but also to facilitate the corresp ondence problem We

have shown that one view can b e transformed toward the second view which is equivalent to

rst pro jecting the ob ject onto the quadric and then pro jecting the quadric onto the second

view For example in the implementation section we have shown that two views of a face

with typical displacements of around pixels are brought closer to around a pixel

displacement by the transformation Most optical ow metho ds can deal with such small

displacements quite eectively

Acknowledgments

Thanks to David Beymer for providing the pair of images used for our exp eriments and to

Long Quan for providing the co de necessary for recovering epip oles

References

JR Bergen and R Hingorani Hierarchical motionbased frame rate conversion Technical

rep ort David Sarno Research Center

OD Faugeras QT Luong and SJ Maybank Camera self calibration Theory and exp eri

ments In Proceedings of the European Conference on Computer Vision pages Santa

Margherita Ligure Italy June

BKP Horn Relative orientation International Journal of Computer Vision

JJ Ko enderink and AJ Van Do orn Ane structure from motion Journal of the Optical

Society of America

BD Lucas and T Kanade An iterative image registration technique with an application to

stereo vision In Proceedings IJCAI pages Vancouver

QT Luong R Deriche OD Faugeras and T Papadop oulo On determining the fundamental

matrix Analysis of dierent metho ds and exp erimental results Technical Rep ort INRIA

France

SJ Maybank The pro jective geometry of ambiguous surfaces Proceedings of the Royal Society

of London

A Shashua Corresp ondence and ane shap e from two orthographic views Motion and recog

nition AI Memo No Articial Intelligence Lab oratory Massachusetts Institute of Tech

nology Decemb er

A Shashua Geometry and photometry in D visual recognition PhD thesis MIT Articial

Intelligence Lab oratory AITR Novemb er

A Shashua On geometric and algebraic asp ects of D ane and pro jective structures from

p ersp ective D views nd European Workshop on Invariants Azores Islands Portugal Octo

b er Also AI Memo No Articial Intelligence Lab oratory Massachusetts Institute

of Technology July

A Shashua and S To elg Quadric reference surfaces in D vision AI Memo No Arti

cial Intelligence Lab oratory Massachusetts Institute of Technology