Polytope samplers for inference in ill-posed inverse problems The Harvard community has made this article openly available. Please share how this access benefits you. Your story matters Citation Airoldi, Edoardo M., and Bertrand Hass. "Polytope Samplers for Inference in Ill-posed Inverse Problems." Proceedings of Machine Learning Research 15 (2011): 110-18. Published Version http://proceedings.mlr.press/v15/airoldi11a.html Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:37304568 Terms of Use This article was downloaded from Harvard University’s DASH repository, and is made available under the terms and conditions applicable to Other Posted Material, as set forth at http:// nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of- use#LAA Polytope samplers for inference in ill-posed inverse problems Edoardo M. Airoldi, Bertrand Haas Department of Statistics, Harvard University Abstract than observations (e.g., sound tracks) available (Lee et al., 1999; Parra and Sajda, 2003); inference on We consider linear ill-posed inverse problems cells of a contingency table where two-way and multi- y = Ax, in which we want to infer many way margins are given (Bishop et al., 1975; Dobra count parameters x from few count observa- et al., 2006), and network tomography (Vanderbei and tions y, where the matrix A is binary and Iannone, 1994; Vardi, 1996; Tebaldi and West, 1998; has some unimodularity property. Such prob- Cao et al., 2000; Medina et al., 2002; Zhang et al., lems are typical in applications such as con- 2003; Liang and Yu, 2003; Airoldi and Faloutsos, 2004; tingency table analysis and network tomog- Lawrence et al., 2006; Fang et al., 2007; Blocker and raphy (on which we present testing results). Airoldi, 2011). Two main approaches to these prob- These properties of A have a geometrical im- lems have been proposed in the literature: sequential plication for the solution space: It is a con- MCMC methods, in which intelligently designed pro- vex integer polytope. We develop a novel ap- posal distributions attempt to explore the space of so- proach to characterize this polytope in terms lutions efficiently, and algebraic geometry methods, in of its vertices; by taking advantage of the ge- which difficult calculations precisely characterize the ometrical intuitions behind the Hermite nor- space of solutions in terms of structured polynomi- mal form decomposition of the matrix A, als (Diaconis and Sturmfels, 1998; Chen et al., 2005; and of a newly defined pivoting operation to Dobra et al., 2006; Dobra and Fienberg, 2008; Dobra travel across vertices. Next, we use this char- et al., 2009). The MCMC based are widely used, but acterization to develop three (exact) poly- approximate. The algebraic methods are exact and tope samplers for x with emphasis on uni- elegant, but already computationally infeasible in low form distributions. We showcase one of these dimensional problems. samplers on simulated and real data. Here, we focus on applications where the matrix A is a 0, 1 -matrix that is unimodular (definition be- { } 1 INTRODUCTION low). Ill-posed inverse problems with such properties arise naturally when sampling contingency tables and Problem settings where we have low dimensional data in network tomography, in which x and y have the and a high dimensional parameter space arise often in additional constraint of being integer-valued. Surpris- model-assisted discovery of mechanistic principles in ingly, in such problems the space of solutions has not the biological and social sciences. We consider linear been characterized. However, it has a simple geomet- ill-posed inverse problems y = Ax with x, y 0, where rical structure: it is an integer convex polytope. ≥ the dimension of y is less than the dimension of x. In this paper we develop polytope samplers, a fresh Given y and A we want to make inferences on x. new approach to inference in ill-posed linear inverse This flavor of ill-posed linear inverse problem arise as problems. The innovation of our approach is two-fold: the core inference tasks underlying a number of appli- (i) we develop new algorithms to identify all the ver- cations, including image super-resolution and positron tices of the solution polytope, and (ii) we develop three emission tomography where we want to combine many strategies to build a sampling distribution on the poly- 2D images in a 3D image consistent with 2D con- tope, as a generalization of the Dirichlet distribution straints (Shepp and Kruskal, 1978; Vardi et al., 1985); over the simplex. We accomplish these tasks by tak- blind source separation where there are more sources ing advantage of the geometry of the problem, which is well specified by the Hermite normal form decom- Appearing in Proceedings of the 14th International Con- position of the matrix A. ference on Artificial Intelligence and Statistics (AISTATS) Proofs of new results are in the online suplementary 2011, Fort Lauderdale, FL, USA. Volume 15 of JMLR: W&CP 15. Copyright 2011 by the authors. material. We provide references for existing results. 110 Polytope samplers for inference in ill-posed inverse problems 2 IDENTIFYING THE SOLUTION This is the Hermite normal decomposition of A, and POLYTOPE the n n square matrix on the left hand side is denoted Q as is× usual in such decomposition. So Q is naturally Definition 1. A square integer matrix is unimodular decomposed into two or four blocks Q =(Q1,Q2)= if it is invertible as an integer matrix (in particular its Q11 Q12 Q11 1 . Here Q1 = , Q11 = A1− , determinant is 1). By extension we define a rectan- Q21 Q22 Q21 gular matrix to± be unimodular if it has full rank and etc.! In particular," it should be! clear from" the Hermite each square submatrix of maximal size is either uni- normal decomposition that the columns of Q2 generate modular or singular (0 determinant). A matrix is to- the null-space of A. tally unimodular (or TUM) if each square submatrix (of any size) is either unimodular or singular (in par- 2.2 Finding a first vertex of the solution ticular entries of a TUM matrix are in 1, 0, +1 ). polytope {− } Our goal is to sample solutions of After defining the operation of pivoting on a matrix (algorithm 1) which is central to our algorithms, we Ax = y, x 0 (1) ≥ restrict our attention to matrices A that are TUM. where A is a given unimodular m n matrix with However, Corollary 0.1 (in sup. mat.), shows that A 0, 1 entries and y is a given integer× positive vector can be only unimodular for our algorithms to work. (solutions{ } to 1 generalize straightforwardly to solutions Lemma 2.2 (Sup. Mat.). If A is TUM, then blocks of the problem where y is non-negative). Q2 and Q12 of Q introduced above are also TUM. 2.1 Integer geometry preliminaries In particular Q12 has entries in 1, 0, +1 . {− } Proposition 2.3 (Sup. Mat.). Given a TUM matrix n Definition 2. A polyhedron of dimension d in R , Q2 as above, a row index i, and a column index j as n d is the intersection of finitely many half spaces input, the algorithm 1 returns a matrix Q2! of the same that≥ is not contained in an affine space of dimension Q12! form than Q2, that is, Q2! = . Moreover, d 1.Apolytope of dimension d is defined as the In m convex− hull of finitely many points not all lying in a ! − " Q2! is also TUM. d 1 affine space. − It is straightforward that the space of solutions to 1 Algorithm 1 Pivot(Q2, i, j) is a polyhedron of dimension n m. That this poly- Q2! [,j] Q2[,j] hedron is bounded (and therefore− a polytope) results for k =1← , . , n excluding k = j do from A having non-negative entries and y being strictly Q! [,k] Q [,k] Q [i, j] Q [i, k] Q [,j] 2 ← 2 − 2 ∗ 2 ∗ 2 positive. Moreover we have: Q! [,j] Q! [k, j] Q! [,j] 2 ← 2 ∗ 2 Lemma 2.1 (Sup. Mat.). The space of real solutions Swap rows i and m + j of Q2! x to equation 1 is an integral polytope. return Q2! Proof. The vertices are the intersections of the affine For short-hand we call a vertex, a vertex of the solu- solution space of Ax = y with the (n m)-coordinate tion polytope. A coordinate m-plane is the set of all planes bording the non-negative orthant.− So a vertex x points with a given set of n m coordinates always has n m zero coordinates. Let’s gather the rest of the zero. Vertices of the solution− polytope are intersec- − coordinates into a positive integer vector x! of dimen- tions of the n m dimensional affine space Ax = y sion m. And let’s gather the corresponding columns with the portions− of coordinate m-planes that border 1 of A into a square matrix A1; so we get the equation the non-negative orthant. Let x! = Q− x, so finding A1x! = y. If A1 was singular, the latter system would solutions to system y = Ax amounts to finding solu- have either none or infinitely many solutions, which tions to AQx! = y, Qx! 0. Since AQ =(Im, 0), the ≥ y would contradict that x is a vertex. So A1 is unimod- system AQx! = y has the obvious solution x! = 0 . 1 ular and x! = A1− y. And since y is integer, x! is also Let y! = Q11y so a solution to Ax = y is x = Qx! = integer. y! # $ (Q1,Q2)x! = Q1y = 0 . Since x has n m zero coordinates it belongs to a co- Since A is unimodular, it has at least one unimod- # $ ordinate m-plane.− However this might not be in the ular submatrix A of maximal size.