A -Based Approach to the Image Moment Problem

JUDIT MARTINEZ, JOSEP M. PORTA and FEDERICO THOMAS Insitut de Rob`otica i Inform`atica Industrial (CSIC-UPC), Llorens Artigas 4-6, 08028 Barcelona, Spain {porta,thomas}@iri.upc.edu

Abstract. An image can be seen as an element of a vector space and hence it can be expressed in as a linear combination of the elements of any non necessarily orthogonal basis of this space. After giving a matrix formulation of this well-known fact, this paper presents a reconstruction method of an image from its moments that sheds new light on this inverse problem. Two main contributions are presented: (a) the results using the standard approach based on the approximation of the result using orthogonal polynomials can also be obtained using matrix pseudoinverses, which implies higher control on the numerical stability of the problem; and (b) it is possible to use basis functions in the reconstruction different from orthogonal polynomials, such as Fourier or Haar basis, allowing to introduce constraints relative to the bandwidth or the spatial resolution on the image to be reconstructed.

Keywords: The moment problem, image moments, moment-invariant image approximations, basis selec- tion.

1. Introduction reconstructing an image from estimates of its mo- ments. In [13], the moment problem has also arisen A very common problem in physics and engineer- when approximating an image to simplify it. In ing is known under the general title of “the mo- this work, a finite number of moments are used to ment problem” [18]. Corresponding to some fi- reconstruct an approximation of the Fourier coeffi- nite number of observations, we are given a set cients of the corresponding image. Unfortunately, of moments –the integrals of various given func- images are not treated as 2D discrete functions. tions with respect to the measure. Since these Instead, using a zigzag scan, they are converted moments will not determine the measure uniquely, into a linear form. the problem consists in deciding which is the best The reconstruction of an image from a set of its estimate. In pure mathematics, this problem dates moments is not necessarily unique. In other words, back to Theodor Stieltjes who proposed it in a pa- it is an ill-posed problem. Therefore, all possible per published in 1894. His work on the moment methods to solve it must impose extra constraints problem was continued and extended primarily by so that the solution becomes unique. Hausdorff and Hamburger. For classical overviews The standard reconstruction method of an image of the subject, including comprehensive historical from some of its moments is based on the least- attributions of classical results, see [1] or [16]. A squares approximation of the image using orthog- more recent survey of the wide range of approaches onal polynomials [17, 15, 12]. Polynomials are the to the problem, including applications, is [5]. most straightforward choice among all possible or- This paper is concerned with the moment prob- thogonal basis functions because they can be eas- lem for images or, more precisely, with the prob- ily related to the multinomial functions that are lem of reconstructing an image from a set of its used to obtain the geometric moments. Legendre geometric moments. and Zernike polynomials were first used in [17]. The moment problem for images arises in sev- They are orthogonal polynomials for continuous eral applications. In [11], the problem of inverting variables in rectangular and polar coordinates, re- the Radon transform is reformulated into that of spectively. However, they are not orthogonal for

1 2 Submitted to the International Journal of Mathematical Imaging and Vision

discrete variables, contrary to what is assumed by matrices are always embraced by parenthesis when some authors [15, 6]. Tchebichef polynomials were superscripts refer to power or transpose. used in [8] and [12] which are orthogonal polynomi- Any discrete image of size a b, say Iab, can als in the discrete domain. Independently of the be seen as a vector in a×b or, alternatively,× as a chosen set of polynomials, the standard method bidimensional functionℜ that maps all the points of assumes null projection coefficients onto the cho- the uniform lattice 1, 2,...,a 1, 2,...,b onto { }×{ } sen polynomial set of order higher than the maxi- real values. Then, Iab can be uniquely expressed mum order of available moments. This solves the as a linear combination of the functions of a basis ill-posesness and the solution becomes unique. In set, i.e., a set containing ab linearly independent order to avoid this assumption, which is difficult bidimensional functions, which will be denoted by kl to interpret in terms of the image properties, a the set Ξab , so that maximum entropy method was proposed in [14]. { } It consists in obtaining the image with maximum a b kl kl entropy with the desired moments. Solving the Iab = α Ξab. k l problem using Lagrange multipliers permits to ob- X=1 X=1 tain an explicit form of the reconstructed image in To avoid in what follows this double summation terms of an exponential function. Alternatively, in the formulation of the problem, we introduce a [10] proposes minimizing the divergence of the im- matrix-based formulation, but first we need some age, instead of maximizing its entropy, using also definitions. a variational approach. Unfortunately, both ap- proaches assume a continuous domain for the im- Definition 1 (Basis matrix). The functions in age. any basis set are assumed to be separable and In this paper, we propose a reconstruction kl equally defined for both coordinates, i.e., Ξab = method that permits introducing constraints that k l t k l φa(φb) , where φa and φb are vectors which can be interpreted in terms of image properties, will be grouped in matrices of the form Φab = such as bandwidth or spatial resolution. We also 1 b (φa,..., φa) called basis matrices. show how the standard least-squares reconstruc- a tion method can be seen as a particular case of Definition 2 (). The matrix Γb = t it. First, we introduce the necessary mathematical (Φab) Φab, containing the inner products between background. Section 3. reformulates the standard the elements of the corresponding basis matrix, is method in terms of the presented formalism. Sec- called a Gram matrix. tion 4. generalizes the result to other orthogonal a k l bases different from polynomials. Finally, section Note that, since Γb [k, l] = φa, φa , the Gram h i 5. contains the conclusions and prospects for fu- matrices are diagonal for orthogonal basis sets and ture research. the identity for orthonormalized basis.

Definition 3 (). The matrix 2. Mathematical background containing the projection coefficients of image Iab kl onto the first m n elements of Ξab are called 2.1. Notation and definitions projection matrices× , which can{ be expressed} as m t Ω Φ t I Φ . Let zm denote a column vector, z its trans- mn = ( am) ab bn ∈ ℜ m pose, and zm[k], with k = 1,...,m, each of its el- kl k t l m×n Note that Ωmn[k, l] = Iab, Ξab = (φa) Iab φb. ements. Likewise, let Zmn denote a ma- h i ∈ ℜ Image Iab can be approximated in terms of the trix of size m n and Zmn[k, l], its element (k, l), × first m n elements of Ξkl by where k = 1,...,m and l = 1,...,n. For simplic- × { ab } ity, square matrices will only have one subscript. m n Superscripts are used to denote any parameter on ˆmn kl kl t Iab = λ Ξab = Φam Λmn (Φbn) , which a matrix depends. Two unary matrix opera- k=1 l=1 tions are used: ( )t denotes the transpose of a given X X −·1 kl matrix; and ( ) , its inverse. To avoid confusions, where m a, n b, and Λmn[k, l] = λ . · ≤ ≤ Submitted to the International Journal of Mathematical Imaging and Vision 3

Definition 4 (Expansion matrix). If the image approximation coefficients λkl are chosen so that the truncation error is minimized using the least- Iab squares error criterion, Λmn is called an expan- sion matrix. mn Iab 2.2. A Theorem

Lemma 1. The approximation of image Iab, in the least-squares sense, can be expressed in terms ˆImn of the projection matrix Ωmn as ab

ˆmn t Iab = Φam Λmn (Φbn) a −1 b −1 t = Φam (Γm) Ωmn (Γn) (Φbn) t −1 = Φam ((Φam) Φam) t −1 t Ωmn ((Φbn) Φbn) (Φbn) Figure 1: Lemma 1 permits to obtain the best approx- mn imation, ˆIab , of image Iab in the least-squares sense, = (Φ )− Ω (Φ )+, am mn bn contained in the subspace represented by the plane in gray, that is, its orthogonal projection onto this sub- where ( )− and ( )+ stand for the left and right space. Theorem 1 is a generalization of this lemma Moore-Penrose· pseudoinverses.· mn that permits to obtain the image Iab contained in kl other subspaces, here represented by a white plane, Proof. Since λ is chosen so that the truncation mn mn that also projects orthogonally onto ˆIab . Both Iab error is minimized according to the least-squares mn and ˆIab preserve the first m × n moments of Iab. error criterion, the subspaces generated by the er- ror and that in which the approximated image is contained are orthogonal. That is, There are infinite images, not only Iab, that lead to the same projection matrix, Ωmn, resulting ij t Ξab, Iab Φam Λmn (Φbn) = 0, from projecting them onto the first m n elements h − i kl × of the basis Ξab . The above lemma permits to for i = 1,...,m and j = 1,...,n. Then, choose from{ this} infinite set the one contained in the subspace spanned by the first m n elements of ij ij t Ξ , Iab = Ξ , Φam Λmn (Φbn) . kl × h ab i h ab i the basis Ξab . The following theorem allows us to select other{ } images contained in spaces spanned Hence, when translating these m n scalar equa- kl by other arbitrary orthogonal basis, Ξ , whose tions into a single matrix equation,× we get ab first m n elements not necessarily span{ the} same × kl t t a b subspace as Ξ . Fig. 1 gives a geometric inter- Ωmn = (Φam) Φam Λmn (Φbn) Φbn = Γ Λmn Γ . ab m n pretation of{ this fact.} Since Gram matrices are obtained from func- Theorem 1. Given the projection matrix Ωmn = tions of a basis set, they are non-singular and this t (Φam) Iab Φbn, the image contained in the sub- proves the Lemma. space expanded by an orthonormal basis, with ba- kl sis matrix Φam, which leads to the same projection Corollary 1. If the basis set Ξ is orthonor- ab matrix Ωmn, is given by mal —we use an overline to distinguish{ } it from the mn t −1 general case— the least-squares approximation of Iab = Φam ((Φam) Φam) the image can be expressed as t −1 t Ωmn ((Φbn) Φbn) (Φbn) mn t a −1 b t −1 t Iab = Φam Ωmn (Φbn) , (1) = Φam (Cm) Ωmn (Cn) (Φbn)

p k l  because Ωmn = Λmn. where C [k, l] = φ ,φ . q h p pi 4 Submitted to the International Journal of Mathematical Imaging and Vision

Proof. Let us consider the following approximated image mn t Iab = Φam Ωmn (Φbn) .

If we want this image to have the same projection m = n = 2 m = n = 4 m = n = 6 m = n = 8 kl coefficients onto Ξ as Iab, then { ab} t t mn Ωmn = (Φam) Iab Φbn = (Φam) Iab Φbn t t = (Φam) Φam Ωmn (Φbn) Φbn. m = n = 4 m = n = 8 m = n = 12 m = n = 16 t t Thus, if (Φam) Φam and (Φbn) Φbn are non sin- Figure 2: Reconstruction of a 8 × 8 pattern (top) and gular, we have that a 16 × 16 pattern (bottom) by imposing null values to t −1 t −1 the unknown moments. Double used. Ωmn = ((Φam) Φam) Ωmn ((Φbn) Φbn) ,

which, when substituted in equation (1), proves and Vpq is a non-square Vandermode matrix the lemma. whose general term is:

l−1 Now, the least-squares approximation given by Vpq[k, l] = k . (5) Lemma 1 can be seen as the particular case of this theorem in which the used orthogonal basis ex- In most applications involving moments, the pands the same subspace as the basis used in the idea is to use the lowest number of moments as projection. possible, that is, max(m, n) min(a, b). As a ≪ Before describing the applications of the above consequence, the matrices Φam and Φbn are sel- theorem, let us reformulate the least-squares dom square and, given Ωmn, there are infinite so- method in terms of the matrix formalism just in- lutions for Iab satisfying (2). troduced. 3.1. A naive approach 3. Revisiting the standard method We can devise a naive reconstruction method by simply assuming that all unknown moments rang- We define the centered geometric moment of order ing from order (m, n) to order (a, b) are zero. (k, l) of image I as ab Then, let us define a b kl k l Ωmn 0m(b−n) µ = (x a/2 ) (y b/2 ) Iab[x,y]. Ω˜ = . (6) − ⌊ ⌋ − ⌊ ⌋ ab 0 0 x=1 y=1 (a−m)n (a−m)(b−n) X X   Then, µkl can be seen as a projection coefficient of According to (2), the image with the moments the image onto a multinomial basis, and the first given by (6) is: m n moments of I can be expressed in matrix ab ˜I = ((Φ )t)−1Ω˜ (Φ )−1. form× as (see Lemma 1 in [9] for details): ab aa ab bb Unfortunately, although the image thus ob- Ω = (Φ )t I Φ mn am ab bn tained preserves the desired moments up to order t t = (Tm) (Vam) Iab Vbn Tn, (2) (m, n), in practice the result has little relation with the original image. Indeed, imposing zero values where to unknown moments leads, in general, to images k−1 l−1 Ωmn[k, l] = µ , (3) with negative pixel values and large variations. In practice, when the dimensions of the original pat- l−1 l−k k−1 ( p/2 ) , if l k, tern and the projection matrix do not coincide, Tp[k, l] = −⌊ ⌋ ≥ (4) (0,  otherwise, the result does not resembles the original. This Submitted to the International Journal of Mathematical Imaging and Vision 5

m = n = 2 m = n = 4 m = n = 6 m = n = 8 m = n = 2 m = n = 4 m = n = 6 m = n = 8

m = n = 4 m = n = 8 m = n = 12 m = n = 16 m = n = 4 m = n = 8 m = n = 12 m = n = 16

m = n = 8 m = n = 16 m = n = 24 m = n = 32 m = n = 8 m = n = 16 m = n = 24 m = n = 32

Figure 3: Reconstructing a 8 × 8 pattern (top), a Figure 4: Reconstructing a 8 × 8 pattern (top) and a 16 × 16 pattern (middle), and a 32 × 32 pattern (bot- 16 × 16 pattern (middle), and a 32 × 32 pattern (bot- tom) from its moments using the approximation de- tom) from its moments using the approximation de- rived from Lemma 1. rived from Theorem 1 and the Tchebichev basis.

phenomenon is exemplified for a 8 8 pattern in Fig. 2-top. If we increase the size× of the pattern matrices of size m m and n n, independently × × (see Fig. 2-bottom), the original image cannot be of the size of the image. reconstructed even in the case that the size of the Fig. 3 presents some examples using IEEE 754 projection matrix and the image coincide. To un- double float representation and Gauss elimination derstand what is the problem, it is enough to real- t inversion. ize that the factorization of (Φaa) involves a Van- dermonde matrix (see equation 2). This matrix is In Fig. 3-top, the reconstruction of a 8 8 pat- × extremely ill-conditioned, so that standard numer- tern is carried out using equation (7). It can be ically stable methods in general fail to compute its seen how the result converges as the order of used inverse accurately, even for moderate sizes. In fact, moments increases. its grows exponentially with its Fig. 3-middle and 3-bottom show the recon- size [2] so that, using a double precision representa- struction of a 16 16 and 32 32 pattern, respec- tion, this matrix cannot be properly inverted using tively. The sizes× of the inverted× Gram matrices a standard inversion algorithm. This is what hap- range from 4 4 to 32 32. When the size of the pened in the example presented in Fig. 2-bottom. image and the× moment× matrix coincide, the re- In sum, this naive approach has little practical constructed image and the original pattern should interest but it has been useful to surface an impor- coincide but, due again to numerical instabilities, tant problem: numerical conditioning. it is not so. Table. 1 shows the maximum size of the projec- 3.2. Using Lemma 1 tion matrices so that the the mean quadratic error a −1 a Using Lemma 1, we have between the and (Γm) Γm, us- ing IEEE double float representation and Gauss ˆmn a −1 b −1 t Iab = Φam (Γm) Ωmn (Γn) (Φbn) (7) elimination inversion, is lower that 0.0001. Images with sizes equal or lower than those maxima can a t where the Gram matrix Γm = (Φam) Φam is be safely reconstructed without resorting to more squared and can be directly inverted. Now, the im- sophisticated inversion algorithms better suited for age approximation is obtained inverting two Gram ill-conditioned systems [4]. 6 Submitted to the International Journal of Mathematical Imaging and Vision

Reconstruction Method m Lemma 1 12 Theorem 1 + Tchebichev 16 Theorem 1 + Fourier 15 Theorem 1 + Haar 8 m = n = 2 m = n = 3 m = n = 4 m = n = 5

Table 1: Size of the projection matrices, m, for which negligible reconstruction error are observed for differ- ent reconstruction methods. In each case, images of m = n = 6 m = n = 7 m = n = 8 m = n = 9 size a × a, with a ≤ m, can be reconstructed with a mean quadratic error lower than 0.0001. These re- sults are obtained using Gauss elimination inversion and IEEE 754 double precision representation. Note that the Haar basis is only defined for m power of 2. m = n = 10 m = n = 11 m = n = 12 m = n = 13

3.3. Using Theorem 1 In Fig. 4, the reconstruction of the same binary pattern as above is carried out using Lemma 2 and m = n = 14 m = n = 15 m = n = 16 kl taking as basis Ξ the normalized Tchebichef { ab } Figure 5: Reconstructing a 16 × 16 pattern from its polynomials [12], so that the corresponding basis moments using the approximation derived from Theo- matrix is rem 1 and the Fourier basis. l tp[k] Φpq[k, l] = , (2l 2)! c(p + l 1, 2l 1) necessary to use an orthogonal basis, contrary to − − − what has been assumed in the literature, because where p the use the Lemma 1 based reconstruction method l yields the same practical results. tl [k] = (l 1)! ( 1)l−j c(p j, l j) p − − − − j=1 X 4. A Novel Reconstruction Method c(l 2 + j, l 1) c(k 1,j 1), − − − − In this section, we explore the possibility that the and c(a, b) is the generalization of the binomial projection and the reconstruction subspaces are numbers given by not the same by applying Lemma 2.

1 if k = 0, j 4.1. Reconstructing a band-limited image c(j, k) =  k if 0 < k j, ≤ While derivatives give information on the high fre- 0 if k >j.  quencies of a signal, moments give information on Note how numerical problems also arise. As in its low frequencies. This duality is clear by realiz- the previous example, Table. 1 indicate the image ing that a one-dimensional real function f(t) can sizes for which the reconstruction can safely done be expressed in terms of its Maclaurin expansion as using IEEE double float representation and Gauss ′′ (n) elimination inversion. ′ f (0) f (0) n f(t) = f(0) + f (0)t + t2 + · · · + t + . . . Since the Tchebichef polynomials and the mo- 2! n! ments basis expand the same subspace, the Corol- (8) lary 2 ensures that the reconstructed image us- and its Fourier transform, say F (w), as n n ing this approximation and that obtained via (jw)2 (−1) (jw) F (w) = m −jw m + m +· · ·+ mn+. . . Lemma 1 are the same (up to numerical instabili- 0 1 2 2 n! ties). Thus, it can be concluded that there is not (9) Submitted to the International Journal of Mathematical Imaging and Vision 7

where mi the moment of order i of f(x). Next, the relationship between moments and low frequencies is made explicit for discrete im- ages. m n m n Fourier coefficients are normally defined as = = 2 m = n = 4 = = 8 m = n = 16 Figure 6: Reconstructing a 16 × 16 pattern from its a b − − − − kl 1 −j2π (x 1)(k 1) + (y 1)(l 1) moments using the approximation derived from Theo- f = I e a b . √ ab rem 1 and the Haar basis. ab x=1 y=1 X X  A relocation of these coefficients in a matrix can and for l > 1 as be carried out so that increasing indexes corre- spond to higher frequency coefficients as follows: + r if s k

can readily used here to find the best basis in fam- ilies of wavelet packet bases or local cosine bases. This is a point that deserves further research. The MatLab implementation developed for the experiments reported in this paper can be down- loaded from http://www-iri.upc.es/people/ porta.

References

[1] N.I. Akhiezer, The Classical Moment Problem and Some Related Questions in Analysis, Hafner Pub- lishing Co., New York, 1965.

[2] B. Beckermann, “The Condition Number of Real Vandermonde, Krylov and Positive Definite Han- kel Matrices,” Numerische Mathematik, Vol. 85, pp. 553-577, 2000. Figure 7: Clockwise from upper left: original image, and reconstructions using pseudoinverses, Haar and [3] R.R. Coifman and M.V. Wickerhauser, “Entropy- Fourier bases, respectively. Data: 256 × 256 image based algorithms for best basis selection,” IEEE subdivided into 16 × 16 blocks; moments up to order Trans. on Information Theory, Vol. 38, No. 2, pp. (4, 4) used. 713-718, 1992.

[4] G.T. Herman, Image Reconstruction from Projec- tions: The Fundamentals of Computerized Tomog- a method for reconstructing an image from a given raphy, Academic Press Inc., New York, 1980. set of moments. None of the former methods pro- vided the proper setting to introduce these con- [5] H.J. Landau, Moments in Mathematics, Short straints. Course Lecture Notes, Vol. 37, Amer. Math. Soc., Images are nonstationary two-dimensional sig- Providence, R.I., 1987. nals with edges, textures, and deterministic ob- jects at different locations. Nonstationary signals [6] S.X. Liao and M. Pawlak, “On image analysis by are, in general, characterized by their local features moments,” IEEE Trans. on Pattern Analysis and Machine Intelligence, Vol. 18, No. 3, pp. 254–266, rather than their global ones. Nevertheless, we 1996. have recovered images by introducing constrains on either its spatial or frequency resolution, which [7] M. Macon and A. Spitzbart, “Inverses of Van- are global constraints. If we want to use local con- dermonde matrices,” The American mathematical straints, we have to simultaneously introduce time Monthly, Vol. 65, No. 2, pp. 95-100, 1958. and frequency constraints. In other words, we need a time-frequency joint representation, such as that [8] J. Mart´ınez, Accumulation Moments. Theory and obtained using a short-time Fourier transform or, Applications, Ph.D. thesis, Technical University of in general, a wavelet transform. The possibilities Catalonia, Spain, 1998. are unlimited and the problem is to find a crite- rion for selecting a basis that is intrinsically well [9] J. Mart´ınez and F. Thomas, “Efficient Computa- tion of Local Geometric Moments,” IEEE Trans. adapted to represent a class of images. By as- on Image Processing, Vol. 11, No. 9, pp. 1102-1112, suming a certain energy distribution on the time- 2002. frequency plane for the image to be recovered from its moments, one can choose or even build a proper [10] P. Milanfar, Geometric estimation and recon- basis that best represents the image with few co- struction from tomographic data, Ph.D. thesis, efficients. Actually, the algorithm presented in [3] Massachusetts Institute of Technology, 1993. Submitted to the International Journal of Mathematical Imaging and Vision 9

[11] P. Milanfar, W.C. Karl, A.S. Willsky, “A moment-based variational approach to tomo- graphic reconstruction,” IEEE Trans. on Image Processing, Vol. 5, No. 3, pp. 459-470, 1996.

[12] R. Mukundan, S.H. Ong, and P.A Lee, “Image Analysis by Tchebichef Moments,” IEEE Trans. on Image Processing, Vol. 10, No. 9, pp. 1357– 1364, 2001.

[13] T.B. Nguyen and B.J. Oommen, “Moment- Preserving piecewise linear approximations of sig- nals and images,” IEEE Trans. on Pattern Anal- ysis and Machine Intelligence, Vol. 19, No. 1, pp. 84-91, 1997.

[14] R.C. Papademetriou, “Reconstructing with mo- ments,” Proc. Int. Conf. on Pattern Recognition, pp. 476-480, 1992.

[15] M. Pawlak, “On the reconstruction aspects of mo- ment descriptors,” IEEE Trans. on Information Theory, Vol. 38, No. 6 pp. 1698–1708, 1992.

[16] J.A. Shohat and J.D. Tamarkin, The Problem of Moments, American Mathematical Society Sur- veys No. 1, 1943.

[17] M.R. Teague, “Image analysis via the general the- ory of moments,” Journal of the Optical Society of America, Vol. 70, No. 8, pp. 920-930, 1980.

[18] D.V. Widder, “The Moment Problem,” chapter 3 in The Laplace Transform, pp. 100-101, Princeton University Press, Princeton, NJ, 1941.