The Geometry of Reflectance Symmetries

Home , Symmetry

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 1 The Geometry of Reﬂectance Symmetries Ping Tan, Member, IEEE, Long Quan, Fellow, IEEE, and Todd Zickler Member, IEEE

Abstract—Different materials reflect light in different restricted to small regions of an image, so that they can ways, and this reflectance interacts with shape, lighting, and be treated as outliers or ‘missing data’. Another approach viewpoint to determine an object’s image. Common materials is to model these non-Lambertian phenomena using para- exhibit diverse reflectance effects, and this is a significant source of difficulty for image analysis. One strategy for metric reflectance models that are more complex than the dealing with this diversity is to build computational tools Lambertian model. This has the important advantage of that exploit reflectance symmetries, such as reciprocity and using all available image data, but it also has a significant isotropy, that are exhibited by broad classes of materials. By limitation: Even relatively simple reflectance models (e.g., building tools that exploit these symmetries, one can create Phong [1]), severely complicate image analysis, and since vision systems that are more likely to succeed in real-world, non-Lambertian environments. In this paper, we develop they are only accurate for limited classes of materials, this a framework for representing and exploiting reflectance approach generally requires new and complex analysis for symmetries. We analyze the conditions for distinct surface each application and each material class. points to have local view and lighting conditions that are A third approach to dealing with complex surface equivalent under these symmetries, and we represent these reflectance is to exploit more general properties. This is conditions in terms of the geometric structure they induce on the Gaussian sphere and its abstraction, the projective the approach that we follow here, and its central thesis plane. We also study the behavior of these structures under is that even though there is a wide variety of materials perturbations of surface shape and explore applications to in the world, there are common reflectance phenomena both calibrated and un-calibrated photometric stereo. exhibited by broad classes of these materials. By building Index Terms—Reflectance symmetry, projective geometry, computational tools to exploit these properties, one can auto-calibration, photometric stereo. create vision systems that are more likely to succeed in real-world environments. One successful example of this I.INTRODUCTION approach is the dichromatic reflectance model [2], which provides the means to exploit the fact that additive diffuse N image of an object is determined through com- and specular reflectance components of non-conducting plex interactions between its reflectance, shape, and A materials are spectrally distinct. surrounding environment. In order to invert this process In this paper, we focus on reflectance symmetries. A and recover scene information, vision systems rely on general BRDF is a function of four angular dimensions— simplified models of image formation, and these often in- two for each of the input and output directions—and clude reduced models of surface reflectance. One common many materials exhibit symmetries over this 4D domain. approach is to assume that surfaces are Lambertian, or Almost all materials satisfy reciprocity ( [3], p. 231), for perfectly matte. According to the Lambertian model, the example, which states that the BRDF is unchanged when bi-directional reflectance distribution function (BRDF) is the input and output directions are exchanged; and many a constant function of the viewing and illumination direc- materials satisfy isotropy, according to which the BRDF tions; and by assuming that surfaces are well-represented is unchanged for rotations about the surface normal of by this simple model, one can build powerful tools for any input/output direction-pair. These symmetries induce stereo reconstruction, shape from shading, motion estima- joint constraints on shape, lighting and viewpoint, and tion, segmentation, photometric stereo, and so on. existing work suggests they can be exploited for visual Most surfaces are not Lambertian, however, so we often tasks such as 3D reconstruction (e.g., [4], [5]). The goal seek ways of generalizing these powerful Lambertian- of this paper is to provide a framework for understanding based tools. One possibility is to assume that non- these symmetries and exploiting them more broadly. Lambertian phenomena, such as specular highlights, are We advocate studying reflectance symmetries in terms P. Tan is with the Department of Electrical and Computer En- of the geometric structures they induce on the Gaussian gineering, National University of Singapore, Singapore 128044. E- sphere. An image of a curved surface under point-source mail:[email protected]. L. Quan is with the Department of Computer Science and En- lighting contains observations of distinct surface points gineering, HKUST, Clear Water Bay, Kawloon, Hong Kong. E- with symmetrically-equivalent local view and lighting mail:[email protected]. directions, and these equivalences induce geometric struc- T. Zickler is with the School of Engineering and Applied Science, Harvard University, 33 Oxford Street, Cambridge, MA 02138. E-mail: tures on the field of surface normals. We show that by [email protected]. representing these structures on the Gaussian sphere and IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 2

bilateral symmetry [6], [11], and reciprocity [12]. The advantage of such symmetry-based approaches is that they avoid the use of low-parameter BRDF models (Lamber- tian, Lafortune [13], Ward [14], Cook-Torrance [15], etc.) that have limited accuracy [16], [17] and often introduce non-linearities that severely complicate vision problems that are already ill-posed. These symmetries are traditionally represented in a local (normal-defined) coordinate system by parameterizing the BRDF domain in terms of halfway and difference Fig. 1. The generalized bas-relief ambiguity [7] is a shape/lighting angles [18]. Accordingly, the complete 4D domain is ambiguity that exists for Lambertian surfaces (left). Reflectance symmetries resolve this ambiguity for any surface that has an additive specular written in terms of the spherical coordinates (θh, φh) of reflection component that is isotropic and spatially-uniform (right). the halfway vector h = (ωi + ωo)/||ωi − ωo||, and those of the input direction with respect to the halfway vector, its abstraction, the projective plane, we obtain concise (θd, φd). Then, the folding due to reciprocity corresponds and intuitive descriptions of the symmetries, as well as to φd → φd+π, and the projection due to isotropy (without convenient tools for applying them to vision problems. bilateral symmetry) is one onto (θh, θd, φd). Bilateral To demonstrate the utility of the proposed framework, symmetry enables the additional folding φd → φd + π/2 we use it to develop new techniques for both calibrated 3 which gives the 3D domain (θh, θd, φd) ⊂ [0, π/2] . and un-calibrated photometric stereo. In uncalibrated photometric stereo, we prove that constraints induced by For vision problems, it is more useful to describe isotropy and reciprocity can resolve the generalized bas- these symmetries in a global, camera-centered coordinate relief ambiguity (Fig. 1) and provide two practical al- system, and that is what we do here. Building on earlier gorithms for doing so. In the calibrated case, we show versions of our work [11], [19], we begin by examining that isotropy and reciprocity constraints can be used to these symmetries on the hemisphere of surface normals recover Euclidean structure from images captured under that is visible to an orthographic camera (i.e., the visible a known, view-centered cone of light sources. This is portion of the Gaussian sphere), and then we show that the achieved by improving the partial reconstruction provided projective plane provides a useful abstraction of the result- by the method of Alldrin and Kriegman [6]. ing structure. In the next, we explore their applications in photometric stereo. II.BACKGROUNDAND RELATED WORK In photometric stereo, we seek to infer three- At an appropriate scale, the reflectance of opaque and dimensional shape from multiple images recorded from a optically-thick materials is described by the bi-directional fixed viewpoint under multiple illuminations. Photometric reflectance distribution function, or BRDF [8]. The BRDF stereo techniques can be either ‘calibrated’ or ‘uncal- is a positive function of four angular dimensions and is ibrated’ depending on whether the lighting conditions written f(ωi, ωo), where ωi and ωo are the directions are known or unknown a priori. The simplest formula- of incident and reflected flux, respectively. As mentioned tion of the problem assumes Lambertian reflectance and in the introduction, many materials exhibit reflectance calibrated acquisition [20]. When the lighting directions symmetries. Reciprocity guarantees that the BRDF is sym- are uncalibrated, the surface shape can only be recov- metric about the input and output directions: f(ωi, ωo) = ered up to certain shape ambiguity [7], [21], [22] even f(ωo, ωi). In many cases, the BRDF is unchanged by when the surface is Lambertian. Many works have been rotations of the input and output directions (as a fixed proposed to investigate when and how this ambiguity is pair) about the surface normal and by reflections of the resolved, which often assume parametric non-Lambertian output direction across the incident (input/normal) plane. reflectance models [23]–[25]. Similar parametric models Materials that satisfy these two symmetries are said to be are also used in calibrated photometric stereo (e.g., [26]– isotropic and bilaterally-symmetric, respectively [9]. It is [30]) to handle general non-Lambertian surfaces. However, also common to use the term isotropic to mean both, and these models are only valid for limited types of surfaces. we will do so here. Further, they are often non-linear and complicate the When one or more of these symmetries is apparent, original problem. In comparison, we apply the geometric radiance measurements that are captured at symmetrically- constraints derived from reflectance symmetries which are equivalent local view and illumination directions must valid for broad class of surfaces. We show these constraints be equal, and this induces joint constraints on shape, can resolve various shape ambiguities in both calibrated viewpoint and illumination. This has been exploited, for and uncalibrated photometric stereo with simple geometric example, for surface reconstruction using isotropy [10], analysis. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 3

d n 2 d d2 m n 2

d1 V S d1 d V S 1 n` m` n`

Fig. 2. On the visible hemisphere of Gauss sphere, two normals form Fig. 3. On the visible hemisphere of Gauss sphere, two normals an isotropic pair if they lie at the intersections of two circles centered form a reciprocal pair if they lie at the intersections of circles that are at source direction s and view direction v. If the BRDF is isotropic (but centered at s and v and whose radii are exchanged. In this example, there otherwise arbitrary) the observed intensity at these points will be equal. are four reciprocal pairs: m ↔ n, m0 ↔ n0, m ↔ n0, and m0 ↔ n. If the BRDF is isotropic and reciprocal, the observed BRDF value (but III.STRUCTUREONTHE GAUSSIAN SPHERE not necessarily the radiance) at these points will be equal. We restrict our attention to surfaces observed by an ideal BRDF is isotropic, the radiance emitted from two points orthographic camera under directional lighting, and we is equal if their normals form an isotropic pair. ignore the global effects of mutual illumination and cast Definition 2. Two surface normals n and m form a shadows. This means that the camera’s measurements of reciprocal pair with respect to source s if and only if they scene radiance can be completely described in terms of satisfy the relative orientation of each local surface patch, and > > > > our analysis can be performed on the Gaussian sphere. For m s = n v and m v = n s. now, we also assume that the surface has uniform material This condition can similarly be interpreted in terms properties so that the BRDF is the same everywhere (this of the intersections of circles centered at s and v. The is relaxed in Sect.VI.) important difference is that the radii of the two circles Consider the orthographic observer to be fixed in direc- are swapped for the two normals. As depicted in Fig. 3, tion v ∈ S2, and choose a global coordinate system so that there are four possible intersections derived from all the z-axis is aligned with this direction (i.e., v = (0, 0, 1)). combinations of circles with centers at s or v and having Let s ∈ S2 represent the illumination direction. (We use two different radii. This family of four normals comprises boldface font to represent unit-vectors throughout this two isotropic pairs ((m, m0) and (n, n0)) and four recip- paper.) Since the great circle through v and s will become rocal pairs ((m, n), (m0, n0), (m0, n) and (m, n0)). Unlike significant, we give it a name and refer to it as the principal isotropic pairs, not every point on the visible hemisphere meridian. has a visible point as its reciprocal partner. This is because n>s < 0 implies m>v < 0, so m is not observed. A. Points and sets of points Using an argument similar to the isotropic case above, it is clear that the local light and view directions at any two We identify sets of surface points having equivalent local normals of a reciprocal pair are such that when a BRDF is light and view directions under isotropy and reciprocity. isotropic and reciprocal, the observed BRDF value at two Definition 1. Two surface normals n and n0 form an points is equal if their normals form a reciprocal pair. It isotropic pair with respect to source s if and only if they is important to realize that this statement does not refer satisfy to emitted radiance, which is the BRDF multiplied by the > n0>s = n>s and n0>v = n>v. foreshortening factor n s. According to our definitions, isotropic pairs are located A geometric interpretation is shown in Fig. 2. Two nor- symmetrically with respect to the plane containing the mals form an isotropic pair if they lie at the intersections principal meridian, and reciprocal pairs are located sym- of two circles centered at v and s on the Gaussian sphere. metrically with respect to the plane perpendicular to the If we consider the local coordinate system for the BRDF principal meridian and passing through the half-vector, domain at each of two such surface normals, it is clear or bisector, of v and s. With the exception of points on that the incoming and outgoing elevation angles (θi and the symmetry planes, most isotropic and reciprocal pairs θo) are the same in both cases. This is because the tetrahe- contain two distinct normals. dron formed by unit vectors {n, s, v} is equivalent under Definition 1 induces an equivalence relation (i.e., one reflection across the principal meridian to that formed by that satisfies reflexivity, transitivity and symmetry), so we {n0, s, v}. Also, if we were to project v and s onto the plane can say that two normals are ‘isotropically equivalent’ if orthogonal to each normal, it is clear that the magnitude they form an isotropic pair. The conditions for a reciprocal of the angular difference between these projections (i.e., pair do not induce an equivalence relation, however, as a |φi −φo|) would also be the same. It follows that when the normal is generally not reciprocal to itself. To form an IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 4 equivalence relation that exploits reciprocity, the condi- on a reciprocal curve, all of its isotropic/reciprocal corre- tions for isotropic and reciprocal pairs must be combined; sponding normals are on the same curve. So it is a union we say that two normals are ‘equivalent under combined of isotropic-reciprocal quadrilaterals. Also, the family of isotropy and reciprocity’ if they form either an isotropic the reciprocal curves partitions the illuminated half of pair or a reciprocal pair. Thinking this way leads to the the Gaussian sphere (i.e., for which s>x > 0) . Finally, following. due to the symmetric arrangement of isotropic-reciprocal quadrilaterals within a reciprocal curve we observe that Definition 3. Four surface normals n, n0, m and m0 form when a BRDF is isotropic and reciprocal, we can choose a a isotropic-reciprocal quadrilateral with respect to source parameterization of a reciprocal curve such that the BRDF s if both (n, n0) and (m, m0) are isotropic pairs and both value along the curve is a symmetric function. One such (n, m) and (n0, m0) are reciprocal pairs. parameterization is given by the azimuthal angle between According to this definition and our previous obser- normal x and the plane perpendicular to the principal vations, it is evident that when a BRDF is isotropic meridian and containing the half-vector. Again, note that and reciprocal, the observed BRDF values at four points this statement involved the BRDF value, which is the is equal if their normals form an isotropic-reciprocal emitted radiance divided by the foreshortening factor n>s. quadrilateral. An example is shown in Figure 8.

IV. STRUCTUREONTHE PROJECTIVE PLANE B. Curves An alternative and arguably more powerful representation It is worth defining two families of curves that are formed of these symmetry-induced geometric structures can be by the equivalence relations described above. constructed by considering the visible hemisphere as a two-dimensional projective plane. As shown in Fig. 4, each Definition 4. On the Gauss sphere, an isotropic curve with > visible normal n = (n1, n2, n3) is considered a point in respect to s is defined as the plane created by a gnomonic (or central) projection > > of the unit hemisphere onto the tangent plane passing α1(s x) + α2(v x) = 0 through the viewing direction v = (0, 0, 1)>. The plane 1 for two constants α1, α2. is equipped with an elliptic metric: the distance between any two points is given by the angular difference between An isotropic curve is a great circle, and it has the the corresponding rays in R3. following properties. First, if a normal is on an isotropic Both points and lines are represented using homoge- curve, its isotropic corresponding normal is also on the nous three-vectors, for which we use the notation x = curve. (This can be easily verified by substituting Defini- (x , x , x ), while noting that x ' αx represent the same tion 1 into the isotropic curve equation.) In other words, an 1 2 3 point for any α ∈ R/{0}. (We use ' to indicate equality isotropic curve is a union of isotropic equivalence classes. up to scale.) As before, we use bold font to represent Second, the family of isotropic curves partitions the Gaus- normalized vectors, so that x = x/||x||. We use the sian sphere. Third and finally, due to the arrangement notation vs , v×s to represent the principal meridian and of isotropic pairs along the curve, when the BRDF is the notation h ' v + s for the half-vector. For illustrative isotropic, the emitted radiance along an isotropic curve purposes, we assume that the source direction s lies in is a symmetric function when the curve is parameterized the upper hemisphere and can be associated with a point by the signed angle to the principal meridian. Figure 7 on the projective plane. This assumption can be easily provides an example. relaxed, however, by applying our analysis to the half- Definition 5. On the Gauss sphere, a reciprocal curve with vector associated with each source direction instead of the respect to s is defined as source direction itself.

> > 2 α1(v x)(s x) + α2(v, s, x) = 0 A. Points and sets of points We first provide definitions for isotropic and reciprocal for two constants α , α .2 1 2 pairs in the projective plane. The equivalence of these Here the notation (x, y, z) = (x × y)>z denotes the definitions is proved in Propositions 7 and 8 of Appendix. scalar triple product. A reciprocal curve has properties that Definition 6. [Alternative to Definition 1] Two surface are analogous to its isotropic counterpart. If a normal is normals n and n0 form an isotropic pair with respect to 1In our previous work [11], this curve is defined as (s>x)+α(v>x) = source s if and only if they satisfy > 0, which cannot include points on the curve v x = 0. (v × s)>(n + n0) = 0, (1) 2In our previous work [11], this curve is defined as (v>x)(s>x) + > > 0 α(v, s, x)2 = 0, which cannot include points on the principal meridian s n s n = . (2) (v, s, x) = 0. v>n v>n0 IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 5

Proposition 10 in Appendix). Notably, this intersection point is independent of m and n. From this we can state the following. Remark 2. Two points n, m on the projective plane form a reciprocal pair if and only if: 1) nm intersect the principal meridian vs at h × (v × s); 2) the join of the bisector of n and m and h (the bisector of v and s) is Fig. 4. The hemisphere of surface normals visible from direction v is represented by a plane obtained by gnomonic projection. This is the real perpendicular to the principal meridian. projective plane, where great circles map to lines and the equator maps to 0 0 a line at infinity. Reciprocity, isotropy, and other reflectance symmetries In Figure 5, both (n, m) and (n , m ) are reciprocal can be studied in terms of the geometric structures that they induce in pairs. Given any point n, its reciprocal correspondence m this plane. can be determined as follows. First, find the point n + m as the intersection of two lines: 1) the join of points n and h×(v×s); and 2) the line through h that is perpendicular to the principal meridian. Then, using the elliptic metric, m is uniquely determined by n and n + m. According to this discussion, the four normals (n, n0, m, m0) comprising an isotropic-reciprocal quadrilateral are such that the principal meridian is the perpendicular bisector of both nn0 and mm0; and both nm and n0m0 intersect the principal meridian at point h × (v × s). The quadrilateral is thus an isosceles trapezoid, and the Fig. 5. In projective plane, an isotropic-reciprocal quadruplet is an isosceles trapezoid centered at h with two sides perpendicular to vs and center of the quadrilateral is the half-vector h (because the two sides intersect at p = h × (v × s). diagonals of the quadrilateral intersect there). This leads to the following. Equations (1) and (2) can be interpreted geometrically Remark 3. Four points n, m, n0 and m0 on the projective as saying that the principal meridian vs is the perpendic- plane form a isotropic-reciprocal quadrilateral if and only 0 ular bisector of the line segment nn . Specifically, Eq. 1 if they form an isosceles trapezoid centered at h, with two 0 says that the mid-point of the line segment nn (given by edges perpendicular to the principal meridian vs and the 03 n + n ) lies on the principal meridian vs, and Eq. 2 says other two edges intersecting vs at h × (v × s). nn0 is perpendicular to vs (see Proposition 9 of Appendix). This leads to the following remark, which is visualized We will refer to the points c = h and p = h × (v × s) Figure 5, where both (n, n0) and (m, m0) are isotropic as the quadrilateral center and quadrilateral peak respec- pairs. tively. Remark 1. Two points n, n0 on the projective plane form an isotropic pair if and only if the principal meridian vs B. Curves is the perpendicular bisector of line segment nn0. The definition of an isotropic curve (Definition 4) can be re-written as > > Definition 7. [Alternative to Definition 2] Two surface (α1s + α2v )x = 0, (5) normals n and m form a reciprocal pair with respect to source s if and only if they satisfy from which it is clear that this curve is a pencil (one- parameter family) of lines in the projective plane. This > > s (n + m) = v (n + m), (3) pencil is formed by the linear combination of the two lines > > > > s (n × m) = v (n × m), (4) l1 = {x | s x = 0} and l2 = {x | v x = 0}, and since the coordinate system is such that v = (0, 0, 1)>, the line Geometrically, Eq. 3 says the line connecting point l is the line at infinity. The intersection of l and l , i.e. n + m and the halfway vector h is perpendicular to the 2 1 2 v ×s, is a common point of all isotropic curves, and since principal meridian (multiply the equation by v>(v + s) = this point is a point at infinity (an ideal point), all isotropic s>(v + s) and apply Proposition 9 of Appendix). Equa- curves are parallel lines in the plane as shown in Figure 6. tion 4 says the line nm intersects the principal meridian The common ideal point v×s is also dual to the principal at the point (v + s) × (v × s) = h × (v × s) (multiply meridian vs, and hence all isotropic lines are perpendicular the equation by v>(v + s) = s>(v + s) and apply to the principal meridian. Hence, a pencil of lines are 3n + n0 is the bisector between n and n0. It is also the middle point isotropic curves if and only if they are perpendicular to of nn0 in the sense of elliptic metric. the principal meridian. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 6

curved surface with uniform, isotropic BRDF under directional illumination. In the remainder of this paper, we explore their application to three-dimensional reconstruction problems, and in particular, to the problem of photometric stereo, where one seeks to infer three-dimensional shape from multiple images recorded from a fixed viewpoint under multiple illuminations. In photometric stereo and some other photometry-based reconstruction problems4, one encounter the situation in which the field of surface normals can be recovered up to a projective ambiguity. That is, instead of the true normal field {nj}, we recover a transformed field {n¯j} that is related to the true field by an unknown linear transform: Fig. 6. Curves of equivalence points in the projective plane. The −> −> black dash line is the line at infinity v. Isotropic curves (red lines) n¯j = A nj/||A nj||, for unknown A ∈ GL(3). are parallel lines perpendicular to the principal meridian, i.e. going In this section, we describe the behavior of our through the common infinite point v × s. Isotropic curves partition the symmetry-induced structures under these transformations. whole projective plane. Reciprocal curves (green conics) are parabolas symmetric about the principal meridian and going through two common We see that the required radiometric relationships between points, s × (v × s) and v × (v × s). Reciprocal curves partition the half isotropic and reciprocal pairs are destroyed by them, and plane lower the line s × (v × s). Points above this line have not visible thus, these radiometric relationships can be used to resolve reciprocal corresponding point. The principal meridian vs is the pole of the infinite point v × s with respect to the reciprocal curves. the reconstruction ambiguity. The definition of a reciprocal curve (Definition 5) is A. General linear transformations equivalent to > > > > As motivation, consider the uncalibrated Lamber- x (α1(vs + sv ) + α2(v × s)(v × s) )x = 0, (6) tian photometric stereo problem, as formulated by which is a pencil of conics in the projective plane. This Hayakawa [21]. Given three or more images of a Lam- pencil is formed by linearly combining the two conics bertian surface under varying, but unknown, directional > > > > c1 = {x | x (vs + sv )x = 0} and c2 = {x | x (v × lighting, the field of surface normals and the source direc- s)(v×s)>x = 0}. These conics pass through two common tions can be recovered up to a 3 × 3 linear transformation points, v × (v × s) and s × (v × s), which can be A that acts according to [21] verified by substitution into Eq. 6. As one of these points, n¯ = A−>n/||A−>n||, ¯s = As/||As||, (7) v×(v×s), is a point at infinity, these conics are parabolas ( [31], p. 117), and these parabolas are symmetric about and inversely as the principal meridian. Since a conic is determined by n = A>n¯/||A>n¯||, s = A−1¯s/||A−1¯s||. (8) five points, and since all reciprocal curves pass through two common points, a reciprocal curve is completely In the projective plane, these can be written much more determined from any three points of an isotropic-reciprocal conveniently as quadrilateral. In short, a pencil of conics are reciprocal −> > −1 curves if and only if they are parabolas symmetric to the n¯ ' A n, ¯s ' As, and n ' A n¯, s ' A ¯s, principal meridian and intersect the principal meridian at where the transformation can now be considered an ele- the two common points (one of them is an infinite point). ment of the projective general linear group; A ∈ P GL(3). Finally, we note that another way of expressing the We consider the scenario in which a normal field {n¯j} link between isotropic and reciprocal curves is through and set of sources {¯si} have been estimated from three or a pole-polar relationship. The common ideal point of all more images {Iij}, and the estimated normals are in the isotropic curves, v×s, is the pole of the principal meridian orbit of the true normals {nj} under group P GL(3). with respect to any reciprocal curve; likewise, the principal meridian is the polar of the common infinite point. This Definition 8. Transformation A preserves isotropic follows from (resp. reciprocal) pairs with respect to s if for all pairs (n, n0), the pair being isotropic (resp. reciprocal) with > > > (α1(vs + sv ) + α2(v × s)(v × s) )(v × s) respect to source s implies the transformed pair (n¯, n¯0) = 2 −> −> 0 = α2(v × s)||v × s|| ' v × s. (A n, A n ) is isotropic (resp. reciprocal) with respect to transformed source ¯si = As. V. TRANSFORMATIONS 4This arises in far-field uncalibrated Helmholtz stereo [32], and it The geometric structures and radiometric constraints de- would arise in Lambertian photogeometric reconstruction [33] if general scribed in previous sections exist whenever one observes a affine cameras were used. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 7

Within the group P GL(3), there are transformations vs is known for at least one image, the rotation ambiguity that preserve isotropic and reciprocal pairs, and there are can be resolved by rotating the perpendicular bisector of others that do not. This means that constraints based on an isotropic pair to align it with vs. Further, the axis of isotropy and reciprocity can be used to restrict the group the mirror reflection is constrained to overlapping with of allowable transformations and reduce the reconstruction vs. This mirror reflection is fully resolved if the principal ambiguity. In the following, we say that two source meridian is known for at least two images. However, the directions are in general position if they are distinct and scaling cannot be resolved from the isotropic structures not coplanar with the view direction. – principal meridian and isotropic pairs, because these structures are preserved by a uniform scaling. Proposition 1. Rotations about the view direction composed with changes in scale and a mirror reflection are the Proposition 2. If the principal meridian is known in only linear transformations that simultaneously preserve at least two images, the identity transformation is the isotropic pairs with respect to two source directions in only linear transformation that simultaneously preserves general position. isotropic-reciprocal quadrilaterals with respect to two sources in general position. Proof: First, from Definition 1 it is easy to verify that isotropic pairs are preserved by all transformations Proof: This proof presents an algorithm to resolve in the group, RSM say, of view-axis rotations composed the general linear ambiguity with the two known princi- with changes in scale. To complete the proof we provide pal meridians and two isotropic-reciprocal quadrilaterals. an algorithm that, given an estimated normal field in the According to the discussion in last paragraph, the general orbit of the true normal field under group P GL(3), finds linear ambiguity can be simplified to a scaling from two a normal field in its orbit under subgroup RSM. isotropic pairs in each image when the principal meridians Suppose we have the ability to determine within the in both images are known. Next, we will show this scaling estimated normal field, pairs of normals that are images can be resolved by an isotropic-reciprocal quadrilateral in under the unknown projectivity of isotropic pairs with one image. Suppose A = diag(λ, λ, 1) is the unknown respect to the true source direction. (Such an ability will scaling. The vertex n of the quadrilateral is transformed 0 to A−>n. In comparison, the light source s is transformed be developed in Sec. VI.) Two such pairs (n1, n1) and 0 to As. This suggests that λ can be determined by a 1D (n2, n2) with respect to a single source direction define a 0 0 search. If we gradually increase λ from 0 to +∞ (λ must point on the line at infinity: n1n1 × n2n2. Thus, from two source directions in general position, two distinct points be positive to ensure the surface facing to the camera), can be located on the line at infinity, and the projective the quadrilateral is constantly expanding. Its center c ambiguity can be reduced to an affine ambiguity by restor- moves away from the origin on vs. However, the source ing the line at infinity (i.e., by finding a transformation that s (and hence the half-vector h) moves toward the origin maps it to (0, 0, 1)). on vs. According to the structure described in Sects. IV, Now, affine transformations preserve the mid-point of a h overlaps with c. Hence, λ can be uniquely determined 0 when these two points meet. line segment. The join of the mid-point of n1n1 with the 0 0 mid-point of n2n2 is a line l perpendicular to n1n1. This 0 provides a pair of perpendicular directions, l and n1n1. B. GBR Transformations From two lighting directions in general position, two such perpendicular pairs can be identified, and this reduces the When a normal field obtained by photometric stereo is affine ambiguity to a similarity which consists of a rota- known to be from a differentiable surface, one can impose tion, translation, scaling and a mirror reflection. Finally, an “integrability constraint” on the normals to reduce the similarities preserve the perpendicular bisector of a line ambiguity. In this case, the normal field can be determined segment. As the original is on the perpendicular bisector up to the generalized bas-relief (GBR) ambiguity, which 0 is represented by linear transformations of the form [7] of n1n1, it can be determined by intersecting two such perpendicular bisectors from two lighting directions non-   1 0 0 coplanar with the view direction. Hence, the translation G =  0 1 0  , µ, ν, λ ∈ R. can be resolved and the similarity is reduced to a rotation µ ν λ about the origin (the view direction), a scale change and a mirror reflection (about an arbitrary line passing the origin In the case of µ = ν = 0, this transformation is known in the projective plane). as the classic bas-relief ambiguity. In the projective plane, Note that this proof provides an 8-normal algorithm an element of the GBR group has the effect of a scale (two isotropic pairs in each of two images) for reducing change by λ composed with a translation by (−µ, −ν). a projective ambiguity to a scale, a mirror reflection and The principal meridian is unchanged by a GBR trans- view-axis rotation. In addition, if the principal meridian formation, so that vs¯ = vs. Hence, a GBR ambiguity is IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 8

Intensity Function BRDF Function

0.8 0.8

0.6 0.6

0.4 BRDF 0.4 Intensity

0.2 0.2

0 0 −90 −45 0 45 90 0 90 180 270 360 Theta (degree) Theta (degree)

Fig. 7. (Left) Green points represent isotropic pairs, which are arranged Fig. 8. (Left) Green points represent reciprocal and isotropic pairs, symmetrically about the principal meridian on an isotropic curve prior which are symmetrically arranged on a reciprocal curve prior to a bas- to a GBR transformation. These points are mapped to the yellow points relief transformation. These are mapped to the yellow points by a bas- by a GBR transformation, and are no longer symmetrically arranged relief transform and are no longer symmetrically located. (Right) The within the curve. (Right) Emitted radiance as a function of signed angle BRDF as a function of position along the reciprocal curves before and from the principal meridian along the isotropic curves before and after after the transformation. This function is initially symmetric, but the the GBR. Since isotropy is not preserved by the GBR, the symmetry in symmetry is destroyed by the bas-relief transformation. the radiance function is lost.

transformation from a reciprocal curve. resolved according to Propositions 2 if isotropic-reciprocal quadrilaterals can be identified in two images. Proposition 5. There are at most four discrete GBR transformations that preserve isotropic-reciprocal quadri- Proposition 3. A GBR transformation maps each isotropic laterals with respect to a source. curve with respect to s ‘as a set’ to an isotropic curve with respect to ¯s. Proof: The proof consists of an algorithm that, given a normal field in the orbit of the true normal field under Proof: Isotropic curves are a pencil of lines perpen- group GBR, finds the true normal field. As shown in Fig- dicular to the principal meridian vs. As the GBR transfor- ure 9, suppose a quadrilateral n¯m¯ m¯ 0n¯0 is identified whose mation is a uniform scaling composed with a translation, pre-image nmm0n0 is an isotropic-reciprocal quadrilateral the set of transformed lines are still perpendicular to under illumination s. The center and peak of this quadri- vs¯ = vs. Therefore, they are isotropic curves with respect lateral are ¯c and p¯ respectively. According to the known to the transformed lighting direction ¯s. transformed lighting direction ¯s, we can also compute the A GBR transformation maps isotropic curves to center and peak as ¯c0 = h¯0 w ¯s + v and p¯0 = h¯0 × (v × ¯s). isotropic curves, but the relative positions of points within The goal is then to find a GBR that maps ¯c to ¯c0 and the curve is not preserved. As a result, it destroys the sym- p¯ to p¯0. However, all of these four points depend on the metry of the intensity function along the curve. This fact is GBR parameters and the lighting and normal directions are demonstrated in Figure 7. This property will be used in the transformed differently, which makes a geometric proof next section to estimate an unknown GBR transformation complicated. from identified isotropic curves by establishing the lost Here, we provide an algebraic approach to proof the intensity symmetry. proposition. By isotropy and reciprocity, an isotropic- 0 0 Proposition 4. A bas-relief transformation maps each reciprocal quadrilateral (n, m, n , m ) with respect to s, reciprocal curve with respect to s ‘as a set’ to a reciprocal satisfies: curve with respect to ¯s. (v × s)>(n + n0) = 0, Proof: Reciprocal curves are a pencil of parabolas s>(n + m) = v>(n + m), symmetric about the principal meridian vs and intersecting s>(n × m) = v>(n × m). it at two common points. The classic bas-relief transfor- > > −1 −1 mation is a uniform scaling by λ, and hence, the set of Substituting n = G n¯/||G n¯||, s = G ¯s/||G ¯s||, we 5 transformed parabolas remain symmetric to vs¯, passing obtain three equations in the unknown parameters µ, ν, λ :

through two common points. Therefore, they are reciprocal (v,¯s, g) = C1, (9) curves with respect to the transformed lighting direction ¯s>(diag(λ2, λ2, 0) + gg>)¯s = C , (10) ¯s. p2 > 2 2 > Similar to the isotropic case, although reciprocal curves ¯s (diag(λ , λ , 0) + gg )(n¯ × m¯ ) = C2(g, n¯, m¯ )(11). ³ 0 ´ > > are mapped as a set by the bas-relief transform, the 1 (v,¯s,n¯) (v,¯s,n¯ ) (¯s n¯)(¯s m¯ ) Here, C = + 0 , C = relative positions of points within the curve is changed. 1 2 v>n¯ v>n¯ 2 (v>n¯)(v>m¯ ) Hence, the intensity symmetry along the reciprocal curve are constants. diag(·) represents a 3 × 3 diagonal matrix, is lost after transformation. This property is illustrated in 5We use the following fact to simplify the derivation. G>a × Figure 8 and later used to estimate the unknown bas-relief G>b/||G>a × G>b|| = G−1(a × b)/||G−1(a × b)||. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 9

Accordingly, we assume that we are given as input a normal field {n¯j} and set of sources {¯si} that have been estimated from three or more images {Iij}, and that the estimated normals are in the orbit of the true normals {nj} under group GBR. In what follows, we present two methods for resolving the GBR ambiguity using isotropy and reciprocity constraints derived from the specular component of the input images. The first follows from Proposition 5 and uses the constraints from a single image, but it has the disadvantage of requiring an exhaustive search over a two-dimensional parameter space. The second follows from Proposition 2 which employs two images instead of one, and has the advantage Fig. 9. In the projective plane, the original isotropic-reciprocal quadruplet is transformed by a scaling followed with a translation. The of requiring an exhaustive search over only one dimension. illumination direction is transformed separately. Constraints from a single image. According to Propo- sition 5, the GBR ambiguity is reduced to a discrete (·, ·, ·) is the triple scalar product of its arguments, and g choice once we identify a quadrilateral n¯m¯ n¯0m¯ 0 whose is the translation vector [−µ, −ν, 1]>. Equations 9-11 are pre-image nmn0m0 is an isotropic-reciprocal quadrilateral linear and quadratic equations about the GBR parameters. with respect to the pre-image of ¯s. It still remains to There are at most four solutions from them 6. discuss how to identify such a quadrilateral in an input This proof provides an one-image algorithm to resolve image. the GBR ambiguity. Similar to the case of multi-view Given an arbitrary normal n¯, we must locate the appro- stereo, where the shape ambiguity is resolved by identify- priate corresponding normals n¯0 and m¯ (computationally, ing the absolute conic, identifying the isotropic-reciprocal m¯ 0 is not required.) These two normals can be located quadrilateral can resolve the ambiguity in photometric sequentially by making use of the fact that the BRDF (or, stereo. in the case of n¯0, the intensity) is equal to that at n¯. We will see that locating the first (isotropic) correspondence VI.APPLICATIONS n¯0 resolves one degree of freedom in the GBR ambiguity, A. Uncalibrated photometric stereo and locating the second (reciprocal) correspondence m¯ resolves the other two. For this discussion, it is helpful to Our objective in this part of the paper is to use isotropy re-parameterize the translational portion (µ, ν) of the GBR and reciprocity to ‘autocalibrate’ photometric stereo within terms of components that are parallel and orthogonal to out the restrictions of low-parameter BRDF models. We the principal meridian. These components are consider the case in which the spatially-varying BRDF of a surface can be represented as linear combination of a 0 ν , s¯1µ +s ¯2ν, (13) Lambertian diffuse component and an isotropic specular 0 component: µ , s¯2µ − s¯1ν. (14)

f(x, y, ωi, ωo) = f1(x, y) + f2(θi, θo, |φi − φo|), (12) Since the GBR includes only translation and scale, the isotropic match n¯0 must lie on the known line that passes where (x, y) denotes a surface point. In this model, through n¯ and is orthogonal to the principal meridian. This the diffuse component varies spatially (i.e., the surface known line is the isotropic curve passing through n¯ and is has ‘texture’), while the specular component is homo- unchanged by the GBR according to the Proposition 3. The geneous. Given three or more images of a surface with match n¯0 can be obtained as the intersection of this line reﬂectance of this form, one can obtain a reconstruction and the isophote on the projective plane (i.e., the equal- up to the GBR ambiguity using existing techniques for intensity contour) that contains n¯. Once n¯0 is determined, diffuse/specular image separation (e.g. [34]), and then one translational component (µ0) is resolved as the value applying uncalibrated Lambertian photometric stereo with that takes the bisector of n¯n¯0 to the principal meridian. integrability [22] to the diffuse component. Next, we locate the reciprocal match, m¯ , and resolve 0 6We can solve for µ, ν from Equation 9-10 as functions of λ2. Then the remaining two degrees of freedom (ν , λ) through we substitute these solutions into Equation 11 to obtain a second order an exhaustive search over this 2D parameter space. A equation about λ2. We can obtain two solutions of λ because it should 0 hypothesis (ν , λ1) yields a hypotheses for the true source, be positive to ensure the surface is visible. Each choice of λ will lead to 1 two possible conﬁgurations of µ and ν. Hence, there are four possible the quadrilateral center and peak: s1, c1 = h1, p1 = solutions in total. h1 × (v × s1). It also yields a hypotheses for the intensity IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 10

(i.e. Equation (9)), we can write this constraint as

s¯2µ − s¯1ν − C1 = 0. (15) Geometrically, this is interpreted as a requirement for symmetry of isotropic pairs about the principal meridian, which is characterized by (v, s, n + n0) = 0. In this way, isotropic constraints from at least two sources in general position can be used to recover both translational parameters, µ, ν, without an exhaustive search. This reduces the GBR ambiguity to a classic bas- Fig. 10. Left: typical example of the objective function for the single- relief ambiguity diag(1, 1, λ), and it remains to determine image method. The existence of local minima at which the function the scale parameter λ. This is consistent with the discus- takes values close to the global minimum (white circle) means that an exhaustive search can be sensitive to noise. Right: typical example of sion following Proposition 1 which claims the classic bas- the objective function for the two-image method. This objective function relief ambiguity cannot be resolved by isotropy. has a clear global minimum. The scale ambiguity can be resolved using reciprocal constraints from a single image. The procedure is much value at m¯ , as like that of the single-image case described above, with the important difference being that it requires only a 1D search m>s n>v n¯>v 0 ¯ ¯ ¯ ¯ −1 0 ¯ over λ instead of a 2D search over (ν , λ). Specifically, I(m) = I(n) > = I(n) > = I(n)λ||G (ν , λ)s|| > . n s n s n¯ ¯s a hypothesis λ1 yields hypotheses for the true source, the quadrilateral center and peak: s1, c1, and p1. It also We exhaustively check all points with this intensity (i.e. generates a hypothesis of the intensity at m¯ . Different from ¯ all points on the same equal-intensity contour as m). For the single-image case, there is no need to check every point ¯ each potential match m1 on this contour, the transformed on the equal-intensity contour. Since the classic bas-relief ¯ ¯ quadrilateral center c2 and peak p2 can be re-estimated ambiguity does not break a reciprocal curve (according by intersecting the perpendicular bisector of n¯n¯0 with the 0 to the Proposition 4), the reciprocal match m¯ must lie line m¯ 1n¯ and the line m¯ 1n¯ respectively. These in turn on the reciprocal curve passing through n¯. Hence, an 0 ¯ ¯ provide an estimation of (ν2, λ2) by mapping c2, p2 back hypothesis of m¯ can be obtained as the intersection of to c1, p1 with a translation and scaling. If the hypothesized 0 0 of this reciprocal curve and an equal-intensity contour in GBR parameters (ν1, λ1) are correct, (ν2, λ2) should be 0 the projective plane. The transformed quadrilateral center consistent with (ν1, λ1). We define the cost of the hypoth- 0 0 0 2 2 ¯c2 and peak p¯2 can be re-estimated by intersecting the esis (ν1, λ1) as the minimum of |ν1 − ν2| + |λ1 − λ2| perpendicular bisector of n¯n¯0 with the line m¯ n¯0 and m¯ n¯ for all m¯ 1 on the equal-intensity contour. The final result 0 respectively as the single image case. These in turn provide of (ν , λ) are obtained by searching a position with the an estimation of λ by the ratio of the distance between minimum cost. 2 c1, p1 and that between ¯c2, p¯2. If the hypothesized value An example of this cost over a large interval of the λ1 is correct, λ2 should be the same as λ1. So we define (ν, λ) parameter-space is shown in the left of Figure 10. 2 a cost of the hypothesis λ1 as |λ1 − λ2| . The final result We only search for positive λ to ensure the surface facing of λ1 is obtained by searching a position with minimum the camera. It should be noted that this method does cost. not guarantee to find a single minimum. As proved in An example of this measure of disagreement over a Proposition 5, there could be 4 discrete solutions. This large interval of λ values is shown in the right of Figure 10. motivates an alternative procedure that uses constraints Unlike the 2D search, we typically see a clear global from two images and eliminates the need for a 2D search, minimum in the objective function at the true value of and relies on a simpler 1D search instead. λ. Constraints from two images. The single-image method Special case of ‘specular-spike’ reflectance. The need described above begins by resolving one translational of exhaustive search in the two described methods comes degree of freedom (µ0 in Eq. 14) using one pair (n¯, n¯0) that from the difficulty of identifying isotropic/reciprocal pairs. can be easily identified as the intersection of the isophote In the special case of ‘specular-spike’ reflectance, a specu- and isotropic curve passing through the point n¯. Given two lar pixel forms isotropic and reciprocal pairs with itself, i.e. images under two sources in general position, the same n¯ = n¯0 = m¯ . Substituting this condition into Equations 9– procedure can be applied in each image to resolve both 11, we find the Equation 11 is degenerated. The first two translational degrees of freedom. equations are still valid. Hence, from a specular pixel 0 Given an image under source ¯si, a pair (n¯i, n¯i) induces in each of two images, we can obtain four equations a linear constraint on the translational parameters (µ, ν) to solve the three GBR parameters. We reach the same IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 11 conclusion as [24] that two specular pixels resolve the GBR ambiguity of ‘specular-spike’ surfaces. Please note that in our formulation µ, ν are solved first in linear equations. In comparison, [24] solves all three parameters together in a nonlinear formulation. The performance of these two methods are further compared in Sects. VII.

B. Calibrated photometric stereo The auto-calibrated approach described in the previous section can be applied whenever the BRDF (or a separable component of it) is isotropic and spatially uniform on the Fig. 11. In calibrated photometric stereo, one component of each surface surface. More general reflectance can be accommodated normal is recovered by exploiting isotropy at each point [6]. Determining when more information is available about the light sources. the remaining degree of freedom at each point can be interpreted as A very general method is that of Alldrin and Kriegman [6], finding an unknown translation of the normal along the line through v which provides a partial reconstruction for surfaces with on the projective plane. isotropic reflectance that varies arbitrarily between surface Gaussian-image of surface points having equal gradient points. Given a set of images I(x, y, t) captured using a magnitude: ||∇z|| = constant. Thus, if we could locate cone of known source directions s(t), t ∈ [0, 2π) centered surface points with normals on such circle, we would about view direction v, this method yields one component recover a surface curve of constant gradient magnitude—a of the normal at every image point (x, y). Specifically, for curve we will refer to as an ‘iso-slope contour’. To get a each pixel it provides the plane spanned by the unknown sense of how this would constrain the surface, consider surface normal and the view direction, but the remaining that when only the iso-depth contours are known, the degree of freedom in each normal cannot be recovered surface can be recovered—at best—up to a differentiable without additional information. In other words, if the function [6]. This is because any two differentiable height surface is differentiable, the surface gradient direction can fields z1(x, y) and z2 = h(z1) will have the same set be recovered at each point, but the gradient magnitude is of iso-depth contours for any differentiable function h(·). unknown. This means that one can recover the ‘iso-depth Here we show that if they also possess the same iso- contours’ of the surface, but that these curves cannot be slope contours, then this arbitrary differentiable function is ordered [6]. reduced to a classic bas-relief transformation (i.e., a linear In this section, we use reflectance symmetries on the scaling of depth). projective plane to reconstruct additional 3D structures. Consider a surface S = {x, y, z(x, y)} that is de- Proposition 6. In the general case, if differentiable height scribed by a height field z(x, y) on the image plane. fields z1(x, y) and z2 = h(z1) are related by a differentiable function h and possess equivalent sets of iso-slope A surface point with gradient zx, zy is mapped via the contours, the function h is linear. Gaussian sphere to point n ' (zx, zy, −1) in the projective plane, and the ambiguity in gradient magnitude Proof: From the functional relationship between from [6] corresponds to a transformation of normal field z1, z2: n¯(x, y) ' diag(1, 1, λ(x, y))n(x, y), where the per-pixel 2 2 2 ||∇z2|| = (∂h/∂z1) ||∇z1|| . scaling λ(x, y) is unknown. As depicted in Figure 11, this can be interpreted as a per-pixel bas-relief transformation, If z1 and z2 possess the same set of iso-slope contours, 2 2 2 where each normal n is translated arbitrarily along the either (∂h/∂z1) is constant or ||∇z1|| = ||∇z2|| = 0 line vn. Now, an isotropic pair (n, n0) has two properties: along each contour. Since sets of iso-depth and iso-slope 0 1) n, n are equally distant from the origin; and 2) lines contours are generically distinct, this implies that ∂h/∂z1 vn and vn0 are symmetric across the principal meridian. is constant and h is linear. The per-pixel transformation destroys the first property but It can be shown that the ‘accidental’ case in which the sets preserves the second. In what follows, we seek to resolve of iso-depth and iso-slope contours are equivalent corre- the shape ambiguity by restoring the former. sponds to a surface-of-revolution with the view direction Having equal distance to v, isotropic pairs lie on circles v as the symmetry axis. centered about the view direction. Any two normals on In light of Proposition 6, it is desirable to be able to such a circle is necessarily an isotropic pair with respect identify the iso-slope contour passing through a given to one of the sources in {s(t)}. Thus, for any normal n, image point. With the partial reconstruction, a normal n¯ the view-centered circle on which it lies can be interpreted is recovered up to a per-pixel bas-relief transformation. as the union of its isotropic matches with respect to the set This transformation does not change the line vn¯, hence, of sources {s(t)}. Now, a view-centered circle is also the the isotropic match n¯0 must lie on the line symmetric IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 12 to vn¯ across the principal meridian. If the surface has uniform reflectance (or has a uniform separable component), the match n¯0 can be located by intersecting this line with the euqal-intensity contour passing through n¯. Such isotropic matches n¯0(t) with respect to all light directions s(t), t ∈ [0, 2π) define the iso-slope contour. This simple method for identifying iso-slope contours could be ambiguous because the equal-intensity contour might intersect the line vn¯0 at two points. This problem is solved by the following extension which also enables us to deal with surfaces having spatially-varying isotropic reflectance of the form:

f(x, y, ωi, ωo)=f1(x, y) + f2(x, y)f3(θi, θo,|φi −φo|), (16) which is a generalization of Eq. 12 that allows a ‘textured’ specular component. Let I(x, y, t) be the recorded radiance, and at each Fig. 12. Resolving the GBR using constraints from reflectance symme- image point (x, y), shift and normalize these observations try. The top row shows linearly encoded normal map. The second and the as third row are the depth maps and surfaces respectively. Columns left to right: calibrated photometric stereo; uncalibrated photometric stereo [22]; (I(x, y, t − φn) − mint I(x, y, t)) our single-image auto-calibration method; our two-image auto-calibration In(x, y, t) , , (17) method. (maxt I(x, y, t) − mint I(x, y, t)) where φn ∈ [0, 2π) is the azimuthal component of the the Cook-Torrance BRDF model. The top row of this surface normal as recovered by [6], and t is extended figure shows a linearly coded normal map, where the RGB periodically: t → t + 2kπ for integer k. Then, if the channels represent x, y, and z components, respectively. spatially-varying BRDF is of the form in Eq. 16, a The second row shows the corresponding surface height necessary condition for two points (x1, y1) and (x2, y2) fields obtained by integrating the normal fields. The third to have normal directions forming an isotropic pair is row shows additional validation by rendering the result In(x1, y1, t) = In(x2, y2, t) ∀ t ∈ [0, 2π). This is because surfaces. From left to right, the columns show results of normalizing the temporal radiance at each pixel to [0, 1] calibrated photometric stereo (i.e., ‘ground truth’); uncali- removes the effects of the spatially-varying reflectance brated photometric stereo [22]; and the results obtained terms f1 and f2. This constraint can be used in the using the single-image method and two-image method matching procedure above by using it in place of the equal- described in Sec. VI-A. For the latter two methods, one intensity contours. or two input images were chosen at random, and one normal n¯ was chosen in each image to provide a ‘seed’ for VII.EXPERIMENTS resolving the three GBR parameters. These results demon- In order to test our methods, we conducted experiments strate that the two proposed auto-calibration methods can with both real and synthetic data. Each set of images was successfully resolve the GBR ambiguity in uncalibrated captured/rendered using directional illumination and an photometric stereo [22], and gives results that are very orthographic camera. In the case of real data, the camera close to the calibrated case. was radiometrically calibrated so that image intensity Figure 13 shows results for experiments on real images. could be directly related to emitted scene radiance. In this figure, the top row shows one of the input images along with the separated diffuse and specular components. A. Uncalibrated photometric stereo The second shows the normal maps as before, and the In order to test the methods for resolving the GBR third and fourth rows show the integrated depth maps and ambiguity, we first captured high dynamic range images surfaces. The columns are arranged in the same way as and separated diffuse and specular reflection components in Figure 12. Again, our auto-calibration methods resolve of the input images using a color-based technique [34], the GBR ambiguity. However, the two-image method often and then applied the method of Yuille and Snow [22] to generates results closer to the ‘ground truth’ than the one- the diffuse images to obtain a reconstruction up to a GBR image method. This can be consistently observed from the ambiguity. We then corrected this ambiguity using both the depth of the ‘fish’, ‘apple’ and ‘pear’ examples. single-image and two-image methods described above. Figure 14 shows the angular errors in recovered normal Figure 12 shows results on a synthetic example using maps with respect to the result from calibrated photometric thirty-six rendered input images of a Venus model using stereo. The first row shows the error of uncalibrated IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 13

Fig. 13. Resolving the GBR using constraints from reflectance symmetry. Top row: one input image with separated diffuse and specular components. Middle row: linearly encoded normal map. Bottom rows: depth maps and surfaces. Columns left to right: calibrated photometric stereo; uncalibrated photometric stereo [22]; our single-image and two-image auto-calibration method, which successfully resolves the GBR ambiguity. photometric stereo. The second and third rows show the with ‘specular spike’ reflectance. As shown in Figure 15, error of our single-image and two-image methods. It is our method generates results closer to the calibrated evident that both methods can significantly reduce the method. We believe this is because our linear formulation error and the two-image method is consistently more (for µ, ν) is more robust than the non-linear equations in accurate than the single-image method. We also observed [24]. several limitations of our method. First, the separation of diffuse and specular reflectance could be inaccurate. When the specular reflection is weak, the specular component has B. Calibrated photometric stereo low signal-to-noise ratio, which causes problems for the In this section, we provide suggestive results for the cal- identification of isotropic and reciprocal structures. This ibrated reconstruction procedure described in Sec. VI-B. explains the large error in the ‘fish’ example. Second, we Given a set of images captured under a known cone of assume the diffuse reflectance to be Lambertian to obtain a source directions, we obtain iso-depth contours (i.e., one reconstruction up to the GBR ambiguity. This assumption component of the normal at each point) by applying the might not be true for certain objects. Last, we require a method of Alldrin and Kriegman [6]. Then, we apply smoothly curved surface so that the geometric structures the method described in Sec. VI-B to compute iso-slope are presented. Our method cannot handle surfaces with curves. only a few discrete normals. Figure 16 shows two sets of results, each using forty We further compared our method with [24] for objects synthetic images rendered using the Cook-Torrance BRDF IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 14

Fig. 14. Surface normal angular difference (in degrees) between. From top to bottom, they are the errors in uncalibrated method, our single-image and two-image method.

Fig. 16. Recovered iso-slope and iso-depth curves. Red and blue Fig. 15. Comparison with the method proposed in [35]. Top rows: curves are the iso-slope and iso-depth curves respectively. These two surface normals and depth. Bottom row: one of the input image and set of curves often do not coincide. Our result is shown on the right. angular errors. Columns left to right: calibrated photometric stereo, For comparison, on the left is ground truth curves obtained from the uncalibrated photometric stereo [25], the method presented in [35], our underlying normal ﬁelds. two-image method. model and forty source directions that are uniformly images, and the helmet example uses 252 input images, distributed along a view-centered cone. On the right in and in each case, the source directions were measured red are shown iso-slope contours that are recovered using during acquisition. Eq. 17 to identify isotropic matches n¯0(t) for the seed points marked in yellow. For comparison, the (generally VIII.DISCUSSION distinct) iso-depth contours recovered by [6] are shown in Isotropy and reciprocity, which are common symmetries blue. The ground-truth curves (computed using the known in the BRDF, induce joint constraints on shape, view- geometry) are shown on the left, and our results match point and lighting that can be used for the radiometric these quite closely. analysis of images. Any image of a surface (convex or Figure 17 shows analogous results on real images. The not) under directional illumination and orthographic view iso-slope curves recovered by our procedure are shown contains observations of distinct surface points having on the right, and as a form of ‘ground truth’, we show local lighting and viewing directions that are equivalent the corresponding contours taken from the complete re- under these symmetries, and these equivalences induce constructions obtained by Alldrin et al. [35]. Despite the geometric structure on the Gauss sphere. We argue that the differences between these two reconstruction procedures projective plane provides a useful abstraction for describ- and their underlying BRDF models, the recovered curves ing this geometric structure and for exploiting it for three- are quite consistent. The gourd example uses 100 input dimensional reconstruction. By developing reconstruction IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 15

the tools we develop can be applied to other tasks. One direction to explore might be photogeometric reconstruction techniques that combine geometric and photometric constraints from images captured under varying viewpoint and lighting. Some of these methods also yield ambiguities that can be described as affine transformations of the normal field [32]. Other directions to consider are reflectometry, illuminant estimation, and compression of appearance data. Finally, while we restrict our attention to reciprocity and isotropy in this paper, the same framework (and the projective plane) can be used to exploit other symmetries as well. In a number of cases, the three-dimensional BRDF domain can be further reduced to a two-dimensional domain because of half-vector symmery [17], [37]. This symmetry provides an even stronger constraint on view, Fig. 17. Recovered iso-slope and iso-depth curves. Red and blue curves are the iso-slope and iso-depth curves respectively. Our result is shown shape, and lighting, and it might be interesting to study on the right. For comparison, on the left is result computed from the in the context of calibrated photometric stereo, where it reconstructions of [35]. could provide conditions for uniqueness that supplement techniques that exploit common BRDF symmetries—and recent empirical results [35]. Conditions for uniqueness only these symmetries—we hope to enable systems that based on parametric BRDF models exist [28], but for the are more likely to succeed in the presence of real-world, most part, conditions that avoid the restrictions of low- non-Lambertian materials. parameter BRDF models have yet to be discovered. We consider applications to both calibrated and uncalibrated photometric stereo, and in the uncalibrated ACKNOWLEDGMENT case, we show that reciprocity and isotropy induce con- We thank the anonymous reviewers for their valuable straints in a single image that are sufficient to resolve the feedbacks and Neil Alldrin for sharing his code and data. generalized bas-relief (GBR) ambiguity. Practically, this This work was supported by the Singapore MOE grant leads to an auto-calibrating reconstruction procedure that R-263-000-555-112 and HOME 2015 project R-263-000- requires only a simple acquisition system (a hand-held 592-305. Long Quan was supported by the Hong Kong light sources) and is likely to succeed for a broad class RGC GRF 619409 and 618510, and the National Natural of objects. Theoretically, it generalizes previous work that Science Foundation of China (60933006). Todd Zickler uses the ‘specular spike’ model of reflectance to resolve was supported by the US National Science Foundation the GBR [23], [24]—that model is a special case of the under Career Award IIS-0546408, the US Office of Naval arbitrary isotropic BRDFs that are considered here. Our Research through award N000140911022, the US Army analysis also provides an alternative to existing methods Research Laboratory and the US Army Research Office that resolve the GBR using parametric BRDF models [25], under contract/grant 54262-CI, and a fellowship from the and it has the relative advantages of working for a broader Alfred P. Sloan Foundation. class of surfaces and allowing precise statements regarding the conditions for uniqueness of reconstructions. REFERENCES The fact that isotropic and reciprocal constraints from a single image are sufficient to resolve the GBR ambiguity [1] B. T. Phong, “Illumination for computer generated pictures,” Com- may help motivate studies of shape perception in the pres- munications of the ACM, vol. 18, no. 6, pp. 311–317, June 1975. [2] S. Shafer, “Using color to separate reflection components,” COLOR ence of non-Lambertian reflectance. Existing perceptual research and applications, vol. 10, no. 4, pp. 210–218, 1985. studies suggest that given a single image of a Lambertian [3] H. v. Helmholtz, Treatise on Physiological Optics, volume 1 ed. surface (left in Fig. 1), different people perceive different Dover (New York), 1925. [4] J. Lu and J. Little, “Reflectance and shape from images using a surfaces, and these surfaces are related to one another collinear light source,” IJCV, vol. 32, no. 3, pp. 1–28, 1999. by GBR-like transforms [36]. The results in this paper [5] T. Zickler, P. Belhumeur, and D. Kriegman, “Helmholtz stereopsis: provide a motive for conducting a similar experiment using Exploiting reciprocity for surface reconstruction,” IJCV, vol. 49, no. 2/3, pp. 215–227, September 2002. non-Lambertian objects with relatively uniform specular [6] N. Alldrin and D. Kriegman, “Toward reconstructing surfaces components (right in Fig. 1). Perhaps peoples’ perceptions with arbitrary isotropic reflectance: A stratified photometric stereo of these surfaces will exhibit less variance. approach,” in Proc. CVPR, 2007. [7] P. N. Belhumeur, D. J. Kriegman, and A. L. Yuille, “The bas-relief While we restricted our attention to applications in ambiguity,” International Journal of Computer Vision, vol. 35, photometric stereo in this paper, it is likely that some of no. 1, pp. 33–44, 1999. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 16

[8] F. E. Nicodemus, J. C. Richmond, J. J. Hsia, I. W. Ginsberg, [34] Y. Sato and K. Ikeuchi, “Temporal-color space analysis of reflec- and T. Limperis, “Geometric considerations and nomenclature for tion,” Journal of Optical Society of America A, vol. 11, no. 11, pp. reflectance,” National Bureau of Standards (US), Monograph 161, 2990 – 3002, November 1994. Oct. 1977. [35] N. Alldrin, T. Zickler, and D. Kriegman, “Photometric stereo with [9] S. R. Marschner, “Inverse rendering for computer graphics,” Ph.D. non-parametric and spatially-varying reflectance,” in Proc. CVPR, dissertation, Cornell University, 1998. 2008. [10] J. Lu and J. Little, “Reflectance and Shape from Images Using a [36] J. Koenderink, A. v. Doorn, A. Kappers, and J. Todd, “Ambiguity Collinear Light Source,” IJCV, vol. 32, no. 3, pp. 213–240, 1999. and the ‘mental eye’ in pictorial relief,” Perception, vol. 30, pp. [11] P. Tan, S. P. Mallick, L. Quan, D. Kriegman, and T. Zickler, 431–448, 2001. “Isotropy, reciprocity and the generalized bas-relief ambiguity,” in [37] F. Romeiro, Y. Vasilyev, and T. Zickler, “Passive reflectometry,” in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2007. Proc. ECCV, 2008. [12] T. Zickler, P. Belhumeur, and D. Kriegman, “Helmholtz stereopsis: Exploiting reciprocity for surface reconstruction,” in Proc. ECCV, PPENDIX 2002. A [13] E. P. F. Lafortune, S.-C. Foo, K. E. Torrance, and D. P. Greenberg, Proposition 7. For v = (0, 0, 1)0 and any other three “Non-linear approximation of reflectance functions,” in SIGGRAPH 0 0 > > 0 ’97: Proceedings of the 24th annual conference on Computer unit vectors s, n, n where n 6= ±n , s n = s n and > > 0 > 0 graphics and interactive techniques. New York, NY, USA: ACM v n = v n , if and only if (v × s) (n + n ) = 0 and Press/Addison-Wesley Publishing Co., 1997, pp. 117–126. s>n s>n0 > = > 0 . [14] G. Ward, “Measuring and modeling anisotropic reflection,” Com- v n v n puter Graphics, vol. 26, no. Annual Conference Series, pp. 265– Proof: We first prove the forward direction. The result 273, 1992. s>n s>n0 [15] R. L. Cook and K. E. Torrance, “A reflection model for computer v>n = v>n0 is trivial since it is simply the quotient of the > 0 graphics,” ACM Transactions On Graphics, vol. 1, no. 1, pp. 7–24, two equations. Next, (v × s) (n + n ) = 0 ⇔ −s2(n1 + 0 Jan. 1982. 0 0 s1 n1+n1 n1)+s1(n2 +n2) = 0 ⇔ = 0 . On the other hand, [16] A. Ngan, F. Durand, and W. Matusik, “Experimental analysis of s2 n2+n2 > > 0 0 > > 0 brdf models,” Eurographics Symposium on Rendering, pp. 117– since n3 = v n = v n = n , we have s n = s n ⇒ 3 0 126, 2005. 0 0 s1 n2−n2 n1s1 + n2s2 = n1s1 + n2s2 ⇔ = − 0 . Hence, [17] M. Stark, J. Arvo, and B. Smits, “Barycentric parameterizations for s2 n1−n1 0 0 isotropic BRDFs,” IEEE T-VCG, vol. 11, no. 2, pp. 126–138, 2005. > 0 n1+n1 n2−n2 2 2 (v × s) (n + n ) = 0 ⇔ 0 = − 0 ⇔ n1 + n2 = [18] S. Rusinkiewicz, “A new change of variables for efficient BRDF n2+n2 n1−n1 02 02 0 representation,” Eurographics Rendering Workshop, vol. 98, pp. n1 + n2 ⇔ n3 = n3, which is true. 11–22, 1998. s>n s>n0 Now, we prove the reverse direction. v>n = v>n0 ⇔ [19] P. Tan and T. Zickler, “A projective framework for radiometric 0 0 > > 0 0 s1 n2n3−n2n3 s n s n s1 n1+n1 image analysis,” in Proc. CVPR, 2009. = − 0 0 and > = > 0 ⇔ = 0 . s2 n1n3−n1n3 v n v n s2 n2+n2 [20] R. Woodham, “Photometric stereo: A reflectance map technique for 0 0 n2n3−n2n3 determining surface orientation from image intensities,” in Proc. Hence, from these two equations we have − 0 0 = n1n3−n1n3 0 SPIE 22nd Annual Technical Symposium, 1978, pp. 136 – 143. n1+n1 0 0 0 0 0 0 ⇔ (n1n3 − n1n3)(n1 + n1) + (n2n3 − n2n3)(n2 + [21] K. Hayakawa, “Photometric stereo under a light source with n2+n2 0 2 0 02 0 0 0 arbitrary motion,” J. Opt Soc. Am., vol. 11, no. 11, 1994. n2) = 0 ⇔ (1−n3)n3 −(1−n3 )n3 +(n1n1 +n2n2)(n3 − [22] A. Yuille and D. Snow, “Shape and Albedo from Multiple Images n ) = 0 ⇔ (n0 − n )(1 + n>n0) = 0. Since n 6= −n0, we Using Integrability,” in Proc. IEEE Conf. Computer Vision and 3 3 3 > 0 0 > s>n s>n0 Pattern Recognition, 1997. have v n = n3 = n3 = v n. Then from v>n = v>n0 [23] O. Drbohlav and R. Sara, “Specularities reduce ambiguity of we immediately obtain s>n0 = s>n. uncalibrated photometric stereo,” in ECCV, vol. 2, 2002, pp. 46–60. [24] O. Drbohlav and M. Chantler, “Can two specular pixels calibrate Proposition 8. Four unit vectors v, s, n, m where n, m photometric stereo?” in ICCV, vol. 2, 2005, pp. 850–1857. non-collinear satisfy s>n = v>m and v>n = s>m, if [25] A. Georghiades, “Incorporating the Torrance and Sparrow > > > model of reflectance in uncalibrated photometric stereo,” and only if s (n + m) = v (n + m) and s (n × m) = Proc. IEEE Int. Conf. Computer Vision, pp. 816–823, 2003. v>(n × m). [26] K. Ikeuchi, “Determining surface orientations of specular surfaces by using the photometric stereo method,” IEEE TRANS. PATTERN Proof: As n, m are distinct, they define a coordinate ANALY. AND MACH. INTELLIG., vol. 3, no. 6, pp. 661–669, 1981. system in R3 along with vector n × m. The result follows [27] S. Nayar, K. Ikeuchi, and T. Kanade, “Determining shape and reflectance of hybrid surfaces by photometricsampling,” IEEE directly from representing v and s in this coordinate Transactions on Robotics and Automation, vol. 6, no. 4, pp. 418– system. 431, 1990. [28] H. D. Tagare and R. J. P. deFigueiredo, “A theory of photometric Proposition 9. Given that v = (0, 0, 1) is the origin in stereo for a class of diffuse non-lambertian surfaces,” IEEE T-PAMI, the 2D projective plane, for any other three points a, b, s vol. 13, no. 2, pp. 133–152, 1991. [29] D. Goldman, B. Curless, A. Hertzmann, and S. Seitz, “Shape on the plane, the line ab is perpendicular to the line vs if and spatially-varying brdfs from photometric stereo,” in IEEE and only if (s>a)/(v>a) = (s>b)/(v>b). International Conference on Computer Vision, 2005. [30] P. Tan, S. Lin, and L. Quan, “Resolution-enhanced photometric Proof: The line ab and vs are a × b and v × s stereo,” in Proc. European Conf. Computer Vision, 2006. respectively. Let the intersection of ab and vs with the [31] H. S. M. Coxeter, The Real Projective Plane, 3rd ed. Springer- Verlag, 1993. line at infinity l∞ = (0, 0, 1) be A , ab × l∞ and [32] T. Zickler, P. Belhumeur, and D. Kriegman, “Toward a stratification B , vs × l∞. Then ab⊥vs if and only if A, B are of Helmholtz stereopsis,” in Proc. IEEE Conf. Computer Vision and harmonic conjugate points with respect to the circular Pattern Recognition, vol. 1, 2003, pp. 548–555. 0 0 [33] N. Joshi and D. Kriegman, “Shape from varying illumination and points I = (1, i, 0) and J = (1, −i, 0) , i.e., the cross viewpoint,” in Proc. IEEE Int. Conf. Computer Vision, 2007. ratio cr(A, B, I, J) = −1. Now, cr(A, B, I, J) = −1 ⇐⇒ IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE, VOL. X, NO. XX, OCTOBER 200X 17

A1B1 +A2B2 = 0, and this reduces to (s1a1 +s2a2)b3 = Long Quan is a Professor of the Department > > > > of Computer Science and Engineering and the (s1b1 + s2b2)a3 ⇐⇒ (s a)(v b) = (s b)(v a) ⇐⇒ > > > > Director of Center for Visual Computing and (s a)/(v a) = (s b)/(v b). Image Science at the Hong Kong University of Science and Technology. He received his Proposition 10. For any four points a, b, s and v, the line Ph.D. in 1989 in Computer Science from INPL, ab intersects sv at the point (v + s) × (v × s) if and only France. He entered into the CNRS (Centre Na- if (s>(a × b))/(v>(a × b)) = (s>(v + s))/(v>(v + s)) 7 tional de la Recherche Scientiﬁque) in 1990 and was appointed at the INRIA (Institut National Proof: The line ab and vs are a × b and v × s de Recherche en Informatique et Automatique) in Grenoble, France. He joined the HKUST respectively. It is clear that (v + s) × (v × s) is a point on in 2001. He works on vision geometry, 3D reconstruction and image- the line vs. This point is the intersection of ab and vs if it based modeling. He supervised the ﬁrst Best French Ph.D. Dissertation is also a point on ab, i.e., (a × b)> ((v + s) × s × v) = 0. in Computer Science of the Year 1998 (le prix de these` SPECIF), the > Piero Zamperoni Best Student Paper Award of the ICPR 2000, and the On the other hand, (v + s) × s × v = s(v (v + s)) − Best Student Poster Paper of IEEE CVPR 2008. He co-authored one v(s>(v + s)). Therefore, (a × b)> ((v + s) × s × v) = 0 of the six highlight papers of the SIGGRAPH 2007. He was elected ⇐⇒ (a × b)>s(v>(v + s)) = (a × b)>v(s>(v + s)) ⇐⇒ as the HKUST Best Ten Lecturers in 2004 and 2009. He has served > > > > as an Associate Editor of IEEE Transactions on Pattern Analysis and (s (a × b))/(v (a × b)) = (s (v + s))/(v (v + s)). Machine Intelligence (PAMI) and a Regional Editor of Image and Vision Computing Journal (IVC). He is on the editorial board of the International Journal of Computer Vision (IJCV), the Electronic Letters on Computer Vision and Image Analysis (ELCVIA), the Machine Vision and Applications (MVA), and the Foundations and Trends in Computer Graphics and Vision. He was a Program Chair of IAPR International Conference on Pattern Recognition (ICPR) 2006 Computer Vision and Ping Tan received the B.S. degree in Applied Image Analysis, is a Program Chair of ICPR 2012 Computer and Robot Mathematics from the Shanghai Jiao Tong Uni- Vision, and is a General Chair of the IEEE International Conference on versity, China, in 2000. He received the Ph.D. Computer Vision (ICCV) 2011. He is a Fellow of the IEEE Computer degree in Computer Science and Engineering Society. from the Hong Kong University of Science and Technology in 2007. He joined the Department of Electrical and Computer Engineering at the National University of Singapore as an assis- tant professor in October 2007. His research interests include image-based modeling, photometric 3D modeling and image editing. He has served on the program committees of ICCV, CVPR, ECCV. He is a member of the IEEE and ACM

Todd Zickler received the B.Eng. degree in honors electrical engineering from McGill Uni- versity, Montreal, QC, Canada, in 1996 and the Ph.D. degree in electrical engineering from Yale University, New Haven, CT, in 2004, under the direction of P. Belhumeur. He joined the School of Engineering and Applied Sciences, Harvard University, Cambridge, MA, as an As- sistant Professor in 2004 and was appointed John L. Loeb Associate Professor of the Natural Sciences in 2008. He is the Director of the Harvard Computer Vision Laboratory, and his research is focused on modeling the interaction between light and materials and developing algorithms to extract scene information from visual data. His work is motivated by applications in face, object, and scene recognition; image-based rendering; content-based image retrieval; image and video compression; robotics; and human-computer interfaces. Dr. Zickler is a recipient of the National Science Foundation Career Award and a Research Fellowship from the Alfred P. Sloan Foundation. His research is funded by the National Science Foundation, the Army Research Ofﬁce, the Ofﬁce of Naval Research, and the Sloan Foundation. He is a member of the IEEE.

7 According to Prop. 9, this is equivalent to the join of (v + s) and (a × b) being perpendicular to vs.