Direct 3D-Rotation Estimation from Spherical Images via a Generalized Shift Theorem

Ameesh Makadia and Kostas Daniilidis ∗ GRASP Laboratory, University of Pennsylvania, Philadelphia, PA 19104 {makadia, kostas}@cis.upenn.edu

Abstract Recent interest from the areas of robotics and graph- ics towards omnidirectional or panoramic sensors has given Omnidirectional images arising from 3D-motion of a cam- rise to a new of approaches for localization which era contain persistent structures over a large variation of are based on the global appearance of the scene. Panoramic motions because of their large field of view. This persis- mosaics or omnidirectional images contain significant scene tence made appearance-based methods attractive for robot content with persistent appearance. Image content can localization given reference views. Assuming that central change only due to change of viewpoint or (dis-)appearing omnidirectional images can be mapped to the , the components but not because of change in viewing direc- question is what are the underlying mappings of the sphere tion. The fact that important structures remain visible en- that can reflect a rotational camera motion. Given such a ables global navigation tasks to be performed image-based. mapping, we propose a systematic way for finding invari- Appearance-based localization [17, 18] has been shown to ance and the mapping parameters themselves based on the be successful by applying PCA to a large set of reference generalization of the Fourier transform. Using results from views. Pajdla and Hlavac [31] studied the rotation just representation theory, we can generalize the Fourier trans- around the optical axis of panoramic systems. Nelson and form to any homogeneous space with a transitively acting Aloimonos [23] have used full spherical views to estimate group. Such a case is the sphere with rotation as the act- motion parameters from spherical motion fields by decou- ing group. The spherical harmonics of an image pair are pling rotations and translations along circular flow fields. related to each other through a shift theorem involving the It is worth revisiting the problem of image transforma- irreducible representation of the rotation group. We show tions now that the scope involves several image modalities how to extract Euler angles using this theorem. We study and that the image deformations are more general. Re- the effect of the number of spherical harmonic coefficients garding conventional imagery, it is basic knowledge that we as well as the effect of violation of appearance persistence can compute affine invariants and that we can compute an in real imagery. affine transformation from combinations of image moments [16, 24] or from Fourier descriptors [32]. Little work has been done on the computation of 3D-rotations from area- 1. Introduction based features [19, 30, 26]. Although there has been much work on projective invariants based on points or curves, Research in localization or estimation of camera displace- there is hardly any work on projective image invariants. ment has been dominated by the detection and tracking of Based on [14, 8, 21] we propose a landmarks or salient features in images. In the vast major- framework for studying image deformations applicable not ity of discrete motion approaches, the features are points. only in the plane but also in other domains like the sphere. Only in the case of differential motion, appearing in the im- This framework involves three steps: age as optical flow, direct approaches have been employed using spatiotemporal image only. The latter ap- • Identify the domain of definition of the signal as a ho- proaches have been quite successful in the case of affine or mogeneous space and the group acting on it. projective transformations in significant areas of images and • particularly in the production of mosaics. Though designed Check whether an irreducible unitary representation for image velocities, their hierarchical implementations can exists for the acting group. Compute the generalized also handle larger motions. Fourier transform of the image.

∗ • Compute the transformation (group action) from a The authors are grateful for support through the NSF-IIS-0083209 and NSF-IIS-0121293 grants as well as through a Penn Research Foundation generalized shift theorem. Compute invariants from grant. the magnitude of the Fourier coefficients.

1 In this paper, we apply this “recipe” on spherical im- energy components we want. We indeed implement and ages arising from omnidirectional cameras with the acting compare with our approach a simple version of the estima- group being the group of rotations SO(3). As a matter of tion of small rotational motion from the brightness change fact there is no simpler action we can model on the sphere, constraint equation. because even the equivalent of a plane shift is a rotation on We will start the paper with the math preliminaries pre- the sphere. Our approach can be used not only for camera senting comprehensively the spherical Fourier transform. motion, but also for template matching and target tracking For a superficial survey of the terms used, the reader is re- on the sphere. The Fourier transform on the sphere is the ferred to the appendix. Then we present the shift theorem well known spherical harmonic transform and the Euler an- and a novel corollary useful for our algorithm in the subse- gles of the group action are computed from the generalized quent section. Experiments are conducted with real images, shift theorem. with artificial or real motion, and our estimation is com- The recipe is known from abstract harmonic analysis and pared to a conventional flow technique. We conclude with generalizes to non-commutative groups. Chirikjian and Ky- an outlook and a short description of the biological rele- atkin [8] have published a very good reference involving vance. engineering applications. A very good introduction can be found in [10]. Previous work in computer vision on spheri- cal harmonics addresses range images [3, 20, 6]. The clos- 2 The Spherical Fourier Transform est approach to ours is that of Burel and Henoco [7] in de- The treatment of spherical harmonics is based on [11, 1] termining the orientation of 3D objects from normalized We parameterize points on the unit sphere S2 using tra- which are obtained from spherical harmonic coef- ditional spherical coordinates, with θ representing the an- ficients. gle of colatitude (θ ∈ [0,π]), and φ the angle of The contribution of this paper to computer vision is (φ ∈ [0, 2π)). Thus, any point η(θ, φ) ∈ S2 can be uniquely twofold: written as the • We present a new methodology on image processing of ( )=(cos sin sin sin cos ) spherical images regarding rotations. Recall that even η θ, φ φ θ, φ θ, θ . simple image shifts on omnidirectional images are ac- We will write f(η) for any function whose domain is re- tually rotations. Note that the sphere is the most conve- stricted to the unit sphere. nient domain to work with. If we use tangential planes The rotation of a point η ∈ S2 is performed by an el- then we cannot capture larger deformations. ement g of the rotation group SO(3). In this paper we • As an example we show direct rotation estimation from will parameterize SO(3) with ZY Z Euler Angles, for rea- the spherical harmonic coefficients. This involves a sons that will be evident later. That is to say, any rotation novel derivate of the spherical harmonic shift theorem g(α, β, γ) ∈ SO(3) can be written as that allows us to express all Euler angles in exponen- ( )= ( ) ( ) ( ) tials. Attitude estimation from global features can be g α, β, γ Rz γ Ry β Rz α regarded as a matching problem and can be tuned by ( ) ( ) choosing any of the high energy spherical coefficients where Rz α and Ry β represent respectively a rotation we prefer. around the z-axis by α radians and a rotation around the y- axis by β radians. For any g ∈ SO(3) and function f(η), How far can such an approach go beyond the sphere and we define the rotation of f(η) with the operator Λg such a 3D-rotation acting on the sphere we do not know. We that −1 know that several results are applicable on the effect of 3D- Λgf(η)=f(g η) rotations on the perspective plane. We discuss these future ( ) ∈ 2( 2) questions in the conclusion. We also define the integration of a function f η L S The closest problem statement to ours is the recovery of as Z collineations directly from image intensities [4, 25, 27] or f(η)dη 2 their derivatives [22]. General differential motions of a cal- η∈S ibrated camera are estimated in [12] and of an uncalibrated where dη =sinθdθdφ is the rotation-invariant volume camera in [5]. The disadvantage of existing direct methods measure on the sphere. Having introduced the notation for is that they rely on a series of iterations involving a warping the integration and rotation of a function f(η) ∈ L2(S2), of the image at every step and in principle they are iterative we proceed to introduce the expansion of functions on the closest point algorithms. Our method would be comparable sphere. to the -based methods when applied hierarchically In traditional Fourier analysis, a basis for all functions but our harmonic decomposition allows us to use any of the on the line is generated by examining the l of the Laplacian restricted to the circle. Similarly, on the The Pmn are generalized associated sphere, the eigenfunctions of the spherical Laplacian (the which can be calculated efficiently using recurrence rela- Laplacian in R3 restricted to the sphere) are the spherical tions. : 2 7→ C ˆg harmonic functions Ylm S . These eigenfunctions Each spherical harmonic coefficient flm of a rotated form an eigenspace of harmonic homogeneous polynomials function Λgf(η) is a of the coefficients which have dimension 2L +1. Thus, the 2L +1spherical of the original function f(η): harmonics for each L ≥ 0 form a basis for any f(η) ∈ S2. Z (2 +1) Λ ˆ = ( 0) l ( 0) ( 0) 0 = −1 The l spherical harmonics of degree l are given as gflm f η Ym gη d η ,η g η η0∈S2 s Z X (2l +1)(l − m)! = ( 0) l ( −1) l( 0) ( 0) l ( )=(−1)m l (cos ) imφ f η Ump g Yp η d η Ym θ, φ Pm θ e 0∈ 2 4π(l + m)! η S |p|≤l X Z = l ( −1) ( 0) l( 0) ( 0) where P l are the associated Legendre Functions Ump g f η Yp η d η m 0∈ 2 |p|≤l η S 2 m + X (1 − ) 2 l m l x d 2 l = ˆ l ( ) P (x)= (x − 1) . flpUpm g . (1) m 2l ! l+m l dx |p|≤l and the normalization factor is chosen to satisfy the orthog- Suppose we are given two spherical images, If and Ig, onality relation where Ig is related to If by some unknown rotation R ∈ Z SO(3). We can use equation (1) to solve for R. 0 l ( ) l ( ) = 0 0 Ym η Ym0 η dη δmm δll , ∈ 2 η S 2.2 Image Invariants where δ is the function. ab We would like to recognize properties of functions that Since spherical harmonics provide an remain invariant under rotations. Specifically, we would for L2(S2), any function f(η) ∈ L2(S2) can be expanded like to identify real-valued functions K(f(η)) such that in this basis: K(f(η)) = K(Λ f(η)) for any g ∈ SO(3).Theshift X X g ( )= ˆ l ( ) theorem allows us to easily identify image invariants. We f θ, φ flmYm θ, φ define the invariant function Kl(f(η)) as l∈N |m|≤l Z X ˆ ˆ ˆ = ( ) l ( ) Kl(f(η)) = flmflm flm f η Ym η dη η∈S2 |m|≤l ˆ ˆ The flm are commonly referred to as the spherical harmonic where flm are the coefficients of f(η).Ifweletfl be the coefficients of f(η). (2l +1)× 1 column vector of all coefficients of degree l we easily verify that K(f(η)) is indeed invariant under ro- ( ) 2.1 The Shift Theorem tations of f η : X ˆ ˆ The shift theorem for functions on the line examines the Kl(Λgf(η)) = ΛgflmΛgflm effect on the Fourier space of translations in the function |m|≤l space. Analogously, for f(η) ∈ L2(S2),wewouldliketo X X X = ( ˆ l )( ˆ l ) study what effect a rotation in the function space has on the flpUpm flpUpm function’s representation in the frequency space. We first |m|≤l |p|≤l |p|≤l ˆ T l l T ˆ note that a rotated spherical harmonic of degree l can be =(fl) U (U ) fl written as a linear combination of spherical harmonics of = K(f(η)) the same degree: X Λ l ( )= l ( ) l( ) gYm η Umn g Yn η , 3. Simplifying rotation estimation |n|≤l Image invariants offer an efficient and accurate technique l where the Umn are the irreducible unitary representations to determine if any two functions on the sphere are rotated of the rotation group SO(3): relative to each other. Once it has been verified that two functions f(η),h(η) ∈ L2(S2) are indeed related by a ro- l ( ( )) = −imγ l (cos( )) −inα ∃ ∈ (3) =Λ Umn g γ,β,α e Pmn β e . tation ( g SO such that f gh), we would like to obtain the Euler angles of the rotation. The method we The number of terms in the summation is l,andsoifwe present here is to utilize the shift theorem (1) in order to create such an equation for each 1 ≤ m ≤ l, we are left retrieve the parameters of the rotation. with a linear system of the traditional form Ax = b,where If we expand (1), we see that the Euler angle correspond- Aisanl×l matrix independent of β, x is a vector dependent ing to a rotation about the y-axis, β, is contained within only on the exponentials of β,andb is a vector dependent l the polynomials Pmn. For the purposes of estimation, the only on the rotated coefficients. Remark: A is full-rank nature of these polynomials will make it quite computa- if there is sufficient spatial orientation in the images. An tionally difficult to accurately extract β. First, we recog- aperture theorem would inform us of the conditions on the nize that any g ∈ SO(3) can be written as g(α, β, γ)= orientations that would lead to singularities in A. π π π π g1(α + 2 , 2 , 0)g2(β + π, 2 ,γ + 2 ). The unitarity of the l representations U gives 3.1 Estimation Algorithm X l ( )= l ( ) l ( ) Umn g1g2 Ump g1 Upn g2 We introduce a method for estimating the rotation g be- 2 2 |p|≤l tween any pair of functions f(η), h(η) ∈ L (S ).Webegin by solving a few simple cases first. We can rewrite (1) as X X ˆ ˆ l π π Λ = ( 1( + 0)) · 3.1.1 Rotation about Z g(α,β,γ)flm flp Upk g α 2 , 2 , | |≤ | |≤ p l k l The first case we will examine is one where the second Eu- l ( ( + π + π )) Ukm g2 β π, 2 ,γ 2 ler angle β is zero. This is equivalent to having only a rota- tion around the Z-axis, and so without loss of generality we to obtain the following: assume α is zero and we only need to estimate γ. We can ˆ ˆg rewrite Corollary 1 as Corollary 1. If flm and flm are coefficients before and af- (3) ˆ −im(γ+ π ) ter the SO action g, they are related as follows Λ f = e 2 C g lm X lm X π g − ( + π ) − ( + π ) −ip( ) ˆ ˆ = im γ 2 ip α 2 ˆ · C = e 2 f · flm e e flp lm lp | |≤ |p|≤l X p l X l (0) l (0) −ik(β+π) ˆ l (0) l (0) −ik(π) ˆ Ppk Pkm e flp Ppk Pkm e flp |k|≤l |k|≤l

It is important to note that the polynomials P l (0) can Since we have only one unknown, we clearly have an over- mn ≥ 1 now be easily precalculated, and that all three Euler angles constrained problem in γ.So,foranyl we choose, we now appear only in exponential terms. We can take advan- have the following minimization problem: X X tage of this corollary to produce the following lemma: π −im(γ+ 2 ) ˆ 2 (e Cpm − Λgfpm) =0 g ˆ ˆ 1≤ ≤ 0≤ ≤ Lemma 2. If flm and flm are coefficients before and after p l m l the SO(3) action g, then for any l>0, l even, we can write This system can be minimized using nonlinear techniques a linear system of the form Ax = b in terms of exponentials such as a quasi-newton approach like Broyden’s method. of the Euler angle β. This method can also be trivially adapted to the case where Proof: We rewrite Corollary 1 as α and β are known. X Λ ˆ = −ipβ l gflm e Cmp 3.1.2 Estimating β |p|≤l − ( + π ) The next case we study is one where α and γ are known and Cl = e im γ 2 · mp X only the rotation around the Y -axis needs to be estimated. − ( + π ) − ( ik α 2 ˆ l (0) l (0) ikπ ) e flkPkp Ppm e However, Lemma 2 presents us with a system that is linear |l|≤l in exponentials of β, so we can simply use a non-linear min- imization technique as described earlier to form an estimate Notice that for an even l,wehaveif|m| odd, |p| even, l = − l | | | | l = for β. then Cmp C(−m)p.If m even, p even, then Cmp Cl . With this we can simplify our equations to get (−m)p 3.1.3 Estimating α, β,andγ ˆg +(−1)m ˆg = X −ipβ( l +(−1)m l ) flm fl(−m) e Cmp C(−m)p If all three Euler angles are unknown, we will use a full non- |p|≤l p odd linear global minimization approach. One simplification we will take is that we will not minimize with respect to all catadioptric images was 2048×1536 pixels without com- three unknowns simultaneously. pression, and the parabolic mirror filled up a region of ap- The first step is to minimize for α and β. For any co- proximately 1400×1400 pixels. The images are mapped to ˆ efficient flm with m =0, we see that Corollary 1 has no the sphere by interpolating onto the θ-φ plane, where sam- dependence on γ. Thus, we can have an overconstrained pling is uniform. Figure 1 shows a sample catadioptric im- system in the two unknowns α and β. We form this sys- age obtained from a parabolic mirror and its corresponding ˆg tem from all equations (derived from Corollary 1) for fp0, projection onto the sphere. p ≤ l,forsomel>1. We can use Broyden’s method to minimize for α and β. Oncewehaveestimatesforα and β, the second step is to estimate γ, as described in section 3.1.1

4 Experiments Figure 1: On the left is a parabolic catadioptric image. In the middle is the spherical image represented on the θ-φ plane, and on In this section we present some results obtained from the the right is the image displayed on the sphere. implementation of our rotation estimation algorithm. In the case where all three Euler angles are estimated, we use an optical-flow based technique for comparison. 4.3 Discrete Spherical Harmonic Transform 4.1 Spherical Images We would like to calculate the spherical harmonic coeffi- cients of a function f(η) using only its samples. Driscoll Catadioptric systems with a unique effective viewpoint have and Healy have shown that if a band-limited function f(η) been proven to be convex reflective surfaces of revolution ˆ has band limit b (flm =0,l > b), then the function can with a parabolic or hyperbolic profile. Geyer and Daniilidis be recovered exactly from the samples, provided the sam- [15] showed that such projections are equivalent to a projec- ples are uniform in θ and φ and there are at least 2b samples tion on the sphere followed by a projection from a point on in each. This allows us to compute the spherical harmonic the sphere axis to the plane. In the parabolic case, the sec- coefficients using only the samples of the original function: ond projection is a stereographic projection from the sphere

2 2b 2b−1 X to the catadioptric plane (also the image plane): ˆ = π X ( ) ( ) ≤ | |≤ flm 2 2 aj f ηjk Ylm ηjk ,l b and m l b =0 =0 θ j k u =cotcos φ 2 = ( ) = πj = πk where ηjk η θj,φk , θj 2b , φk b ,andtheaj θ v =cotsin φ. are weights. These coefficients can be computed efficiently 2 with an FFT along φ. For more information, readers are referred to Driscoll and Healy [11, 1]. Given a calibrated camera and catadioptric image I(u, v) we define its inverse stereographic mapping onto the sphere as 4.4 Comparison method ( ) def= (cot θ cos cot θ sin ) IS θ, φ I 2 φ, 2 φ . We also implement a rotation-estimation technique based on optical flow for purposes of comparison. To calculate This mapping allows us to interpolate an image on the the flow on the sphere, we use the technique described in sphere given a catadioptric image. The range of this map- [9]. The flow equation is ping is only limited by the field of view of the original cata- ◦ 1 dioptric system, and so to fully image the sphere a 360 Iθp˙θ + Iφp˙φ = −It, field of view catadioptric system would be required. sin θ

where Iθ, Iφ,andIt are the spatial (in the θ,φ directions), and temporal image derivatives. The flow vector on the 4.2 Image aquisition T plane tangent to the sphere is (˙pθ, p˙φ, 0) .Ifp is a point To obtain spherical images, we used a catadioptric system on the sphere and p˙ is the flow at point p in cartesian coor- consisting of a Nikon Coolpix 995 digital camera along with dinates, then we have p˙ = ω × p,whereω is the angular a parabolic mirror attachment produced by Remote Reality. velocity about the rotation axis. After calculating the flow The mirror’s field of view is 212◦. The size of the original on the sphere, we have an overconstrained system for the unknown ω. If we assume the the flow is small, we have R = I +ˆω,whereR = g(α, β, γ) and ωˆ is the skew- symmetric matrix corresponding to ω.

4.5 Results Figure 3: Clutter Images. On the left is a reference frame, and on We first present the results of our estimation algorithm ap- the right is the same image with a human in the scene. plied to artificially rotated spherical images. Figure 2 gives an example of a spherical image and its synthetically rotated counterpart. The first test we perform is the estimation of all thee Euler angles using the technique described in sec- tion 3.1.3. In this case the rotation is known, and so we can easily verify the accuracy of our estimation. We tested the algorithm with three different degrees of spherical harmon- ics (i.e. the system minimized only consisted of harmonics 5% 9% 15% up to a certain degree). Table 1 presents the estimated rota- tion angles. Figure 4: Clutter images thresholded to display quantity of clutter. Percentages refer to the number of pixels affected.

Angle l ≤ 5 l ≤ 8 l ≤ 16 α =18◦ 0.64◦ 0.46◦ 0.91◦ β =16.3◦ 1.31◦ 0.70◦ 0.85◦ γ =15◦ −0.37◦ −0.11◦ −0.23◦ Figure 2: Rotated Images. On the left is the spherical image gen- Table 2: Rotation errors from running non-linear minimization erated by the catadioptric system. On the right is the same image over all three angles for images with scale variation. synthetically rotated by g(α, β, γ)=g(60◦, 45◦, 270◦).

Angle l ≤ 5 l ≤ 8 l ≤ 16 α =60◦ 0.12◦ 0.98◦ 0.22◦ β =45◦ 0.06◦ 0.03◦ 0.13◦ γ =25◦ −0.09◦ 0.12◦ 0.10◦ Figure 5: Rotated Images. Two spherical images from catadioptric Table 1: Rotation errors from running non-linear minimization images rotated relatively about the mirror’s optical axis. over all three angles for artificially rotated images. l value implies that no coefficients with degree greater than l were used.

We now study the estimation method’s response to scal- ings of the image. Table 2 shows how accurately the rota- tion is estimated when the scale factor of 10% is used. In addition to scalings, we also examine the effect on our estimation technique when we violate the assumption Figure 6: Real rotated images on the sphere (small rotation). of appearance persistence, which is modeled with clutter in the scene. Figure 3 shows two images, with the only differ- about its optical axis is equivalent to having only one non- ence being a human has entered the scene in the cluttered zero Euler angle (Figure 5). We estimate this single angle image. The rotation is estimated between the original ref- using the technique described in section 3.1.1, and some re- erence frame and a cluttered image which has been rotated. sults are shown in Table 4. Table 3 displays results for varying degrees of clutter. To estimate all three Euler angles we use directly the We now present results from images aquired by actual technique explained in section 3.1.1. Ground truth was cal- rotations of the mirror. Since the parabolic mirror does not culated with physical measurements and verified by fitting image an entire sphere, rotations of the mirror naturally re- the global rotation to images. Results are in Table 5. It is flect appearance of clutter in the scene. Rotating the mirror important to note that for small rotations in real data, the Angle l ≤ 8 l ≤ 16 Flow l ≤ 8 l ≤ 16 Flow l ≤ 8 l ≤ 16 Flow α =7.0◦ −0.23◦ −0.46◦ −0.83◦ 1.3◦ 1.9◦ 1.8◦ ∞ 3.2◦ −2.6◦ β =9.0◦ 0.58◦ 0.20◦ 0.83◦ −0.09◦ −0.17◦ 0.5◦ ∞ 13.7◦ −9.4◦ γ =12.0◦ 0.19◦ 0.14◦ 0.34◦ 1.6◦ 1.4◦ 1.2◦ ∞ −16.1◦ −3.7◦

Table 3: Rotation errors from cluttered estimation. The first column is the ground truth. Columns 2-4 are errors for the first image (5% clutter), columns 5-7 are for 9% clutter, and the remaining columns are for 15% clutter. ∞ means the algorithm did not converge.

Angle l ≤ 5 l ≤ 8 l ≤ 16 γ =30◦ −0.04◦ −0.01◦ 0.02◦ eral motions which still are group actions. Such ones have γ =36◦ 0.06◦ −0.07◦ −0.19◦ to obey to a global depth description, a good example would γ =45◦ −0.98◦ −0.35◦ −0.29◦ be mappings on the sphere analogous to collineations. We believe that configurations involving omnicams on air vehi- Table 4: Rotation errors from estimating γ as described in 3.1.1. cles or mobile robots could thus be covered so that localiza- Three different rotations were tested. tion happens just appearance-based using some reference views. Angle l ≤ 5 l ≤ 8 l ≤ 16 Flow α = −2.4◦ 0.2◦ 0.1◦ 0.2◦ −0.3◦ A Informal math exposition β =9.3◦ −2.1◦ −1.7◦ −2.0◦ −1.2◦ γ =2.2◦ 0.3◦ 0.2◦ 0.5◦ −0.3◦ This section is not meant as a formal enumeration of the assumptions we make. It is a rather intuitive description of Table 5: Rotation errors for real data with small rotation. what is required to apply the recipe of a generalized Fourier Euler angle β is weakly estimated. This is in part due to transform. We assume that the reader is familiar with the the nature of our catadioptric camera system. Since the concept of a group. Let us live with an intuitive definition parabolic mirror’s field of view is 212◦, the spherical im- of a that its elements are on a smooth age is zero-padded beyond the Parallel of θ = 106◦.A and that the group operation and the inversion are smooth 1 general rotation would extend the signal beyond the Paral- maps [2] . The real line and the circle are Lie groups with lel, but not in the case of our camera system. A similar respect to addition and well known matrix Lie groups are phenomenon occurs at the north pole of the sphere. Also, the general linear group of square invertible matrices, the in comparison of estimation techniques, calculation of the rotation groups SO(n), and the . optical flow on the sphere is rather computationally expen- A group G is acting on a space X when there is a map × → sive, whereas very few spherical harmonic coefficients are G X X such that the identity element of the group needed in the approach presented here. leaves X as is and group operations are associative. For example, the isometry group SE(2) acts on the plane R2. The rotation group SO(3) can act on the sphere S2.The 5. Conclusion set of all gx ∈ X for any g ∈ G is called the orbit of x. ∈ There is significant evidence from biological findings, par- If the group possesses an orbit, that means for any a, b = ∈ ticularly in insects [13, 29], that motion perception and nav- X, ga b for a g G, then the group action is called igation are based on global matched filters implemented by transitive. For example, there is always a rotation mapping neurons of almost spherical receptive fields. Our approach one point on the sphere to another. If a H of G ∈ is not a model of such an architecture but provides the tools fixes a point x X then H is called the isotropy group. A (2) for analyzing whether it is possible. In particular, it natu- typical example of an isotropy group is the subgroup SO (3) rally contains the notion of scale in the order of the spheri- of SO acting on the north-pole of a sphere. cal harmonic coefficients. The results are very encouraging A space X with a transitive Lie group action G is called in particularly regarding the number of coefficients neces- a homogeneous space. If the isotropy group is H,itisde- R2 sary to recover the unknown attitude. We might speculate noted with G/H. The plane is the homogeneous space (2) (2) S2 that such coefficients are global encoders of the image and SE /SO . The sphere is the homogeneous space (3) (2) that rotations can be estimated from a simple look-up table SO /SO . associating coefficient combinations with angles. Images are usually defined on homogeneous spaces and their deformations are the group actions. The question In this article, we have treated only the pure rotation is now, for which groups does a Fourier transform exist? case. Though the main apparent motion in omnidirectional The answer requires us first to be able to integrate on the images is due to rotation and our algorithm is robust to exis- tence of small translation, it is interesting to extend it to gen- 1We only miss the condition that it has to be a topological group. group and on the homogeneous space and second to find try. International Journal of Computer Vision, 43:223–243, the Fourier basis analogous to eixω on the real line. For in- 2001. tegration, we ask the group to be unimodular. Without go- [16] E. Hall. Computer Image Processing and Recognition. Aca- ing into Haar measure existence, we will just mention the demic Press, 1979. fact that SO(3) is compact and that compact groups are uni- [17] M. Jogan and A. Leonardis. Robust localization using modular. The of a group exists if the group eigenspace of spinning-images. In IEEE Workshop on Om- possesses an irreducible unitary representation where irre- nidirectional Vision, Hilton Head, SC, June 12, pages 37–46, ducible can be intuitively explained as not having trivial in- 2000. variant subspaces [28]. Among other properties, if consid- [18] A. Kelly. Mobile robot localization from large-scale ap- ered as a basis it is orthogonal. We will use the irreducible pearance mosaics. Intern. Journal of Robotics Research, representation of SO(3) which is the basis of the general- 19:1104–1125, 2000. l (cos ) ized Legendre polynomials Pmn x . [19] R. Lenz. Rotation-invariant operators. In Proc. Int. Conf. on Pattern Recognition, pages 1130–1132, Paris, France, Sept. 28-3, 1986. References [20] A. Matheny and D. B. Goldgof. The use of three and four- [1] G.B. Arfken and H.J. Weber. Mathematical Methods for dimensional surface harmonics for rigid and nonrigid shape Physicists. Academic Press, 1966. recovery and representation. IEEE Trans. on Pattern Analy- [2] A. Baker. Matrix Groups. Springer-Verlag, 2002. sis and Machine Intelligence, 17(10):967–981, 1995. [3] D. Ballard and C. Brown. Computer Vision. Prentice-Hall, [21] M.I. Miller and L. Younes. Group actions, homeomor- Englewood-Cliffs, NJ, 1982. phisms, and matching: A general framework. International [4] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hingorani. Hi- Journal of Computer Vision, 41:61–84, 2001. erarchical model-based motion estimation. In Proc. Second [22] S. Negahdaripour and B.K.P. Horn. Direct passive naviga- European Conference on Computer Vision, pages 237–252, tion. IEEE Trans. Pattern Analysis and Machine Intelligence, Santa Margerita, Italy, May 23-26, G. Sandini (Ed.), Lecture 9:168–176, 1987. Notes in Computer Science 588, Springer-Verlag, Berlin et [23] R. C. Nelson and J. Aloimonos. Finding motion parame- al., 1992. ters from spherical motion fields. Biological Cybernetics, [5] T. Brodsky and C. Fermuller. Self-calibration from im- 58:261–273, 1998. age derivatives. International Journal of Computer Vision, [24] J. Sato and R. Cipolla. Extracting group transformations 48:91–114;, 2002. from image moments. Computer Vision and Image Under- [6] Th. Bulow. Spherical diffusion for 3d surface smoothing. In standing, 73:29–42, 1999. Proc. 1st International Symposium on 3D Data Processing, [25] H. S. Sawhney and R. Kumar. True multi–image alignment Visualization, and Transmission, Padova, Italy, 2002. and its application to mosaicing and lens distortion correc- [7] G. Burel and H. Henoco. Determination of the orientation tion. In IEEE Conf. Computer Vision and Pattern Recogni- of 3d objects using spherical harmonics. Graphical Models tion, pages 450–456, Puerto Rico, June 17-19, 1997. and Image Processing, 57:400–408, 1995. [26] J. Segman, J. Rubinstein, and Y. Zeevi. The canonical co- [8] G.S. Chirikjian and A.B. Kyatkin. Engineering Applications ordinates method for pattern deformation: Theoretical and of Noncommutative Harmonic Analysis: With Emphasis on computational considerations. IEEE Trans. Pattern Analysis Rotation and Motion Groups. CRC Press, 2000. and Machine Intelligence, 14:1171–1183, 1992. [9] K. Daniilidis, A. Makadia, and T. B¨ulow. Image processing [27] H.-Y. Shum and R. Szeliski. Systems and experiment paper: in catadioptric planes: Spatiotemporal derivatives and opti- construction of panoramic image mosaics with global and cal flow computation. In Workshop on Omnidirectional Vi- local alignment. International Journal of Computer Vision, sion, Copenhagen, June 22, pages 3–12, 2002. 36:101–130, 2000. [10] A. Deitmar. A first course in harmonic analysis. Springer [28] B. Simon. Representations of finite and compact groups. Verlag, 2001. Grad, Studies in Math. AMS, 1996. [11] J.R. Driscoll and D.M. Healy. Computing fourier trans- [29] M.V. Srinivasan. Ants match as they march. Nature, forms and convolutions on the 2-sphere. Advances in Applied 392:660–661, 2001. , 15:202–250, 1994. [30] M. Tanaka. On the representation of the projected motion [12] C. Fermuller and J. Aloimonos. Direct perception of three- group in 2+1d. Pattern Recognition Letters, 14:871–678, dimensional motion from patterns of visual motion. Science, 1993. 270:1973–1976, 1995. [31] T.Pajdla and V.Hlavac. Zero phase representation of [13] M.O. Franz and H.G. Krapp. Wide-field, motion- sensitive panoramic images for image based localization. In Int. Conf. neurons and matched filters for optic flow fields. Biological on Computer Analysis of Images and Patterns, pages 550– Cybernetics, 83:185–197, 2000. 557, Ljubljana, Slovenia, Sept. 1-3, 1999. [14] J.-P. Gaulthier, G. Bornard, and M. Silberman. Motion and [32] C.T. Zahn and R.Z. Roskies. Fourier descriptors for plane pattern analysis: harmonic analysis on motion groups and close curves. IEEE Trans. Computers, 21:269–281, 1972. their homogeneous spaces. IEEE Trans. Systems, Man, and Cybernetics, 21:159–172, 1991. [15] C. Geyer and K. Daniilidis. Catadioptric projective geome-