OPTIMIZATION MODELS for SHAPE-CONSTRAINED FUNCTION ESTIMATION PROBLEMS INVOLVING NONNEGATIVE POLYNOMIALS and THEIR RESTRICTIONS by DAVID´ PAPP
Total Page:16
File Type:pdf, Size:1020Kb
OPTIMIZATION MODELS FOR SHAPE-CONSTRAINED FUNCTION ESTIMATION PROBLEMS INVOLVING NONNEGATIVE POLYNOMIALS AND THEIR RESTRICTIONS by DAVID´ PAPP A Dissertation submitted to the Graduate School-New Brunswick Rutgers, The State University of New Jersey in partial fulfillment of the requirements for the degree of Doctor of Philosophy Graduate Program in Operations Research written under the direction of Dr. Farid Alizadeh and approved by New Brunswick, New Jersey May 2011 ABSTRACT OF THE DISSERTATION Optimization models for shape-constrained function estimation problems involving nonnegative polynomials and their restrictions by DAVID´ PAPP Dissertation Director: Dr. Farid Alizadeh In this thesis function estimation problems are considered that involve constraints on the shape of the estimator or some other underlying function. The function to be estimated is assumed to be continuous on an interval, and is approximated by a polynomial spline of given degree. First the estimation of univariate functions is considered under constraints that can be represented by the nonnegativity of affine functionals of the estimated function. These include bound constraints, monotonicity, convexity and concavity constraints. A general framework is presented in which arbitrary combination of such constraints, along with monotonicity and periodicity constraints, can be modeled and handled in a both theoretically and practically efficient manner, using second-order cone programming and semidefinite programming. The approach is illustrated by a variety of applications in statistical estimation. Wherever possible, the numerical results are compared to those obtained using methods previously proposed in the statistical literature. Next, multivariate problems are considered. Shape constraints that are tractable in the uni- variate case are intractable in the multivariate setting. Hence, instead of following the univariate approach, polyhedral and spectrahedral (semidefinite) approximations are proposed, which are obtained by replacing nonnegative splines by piecewise weighted-sum-of-squares polynomial splines. A decomposition method for solving very large scale multivariate problems is also investigated. In order to include more constraints in the same unified framework, the notion of weighted-sum- of-squares functions is generalized to weighted-sum-of-squares cones in finite-dimensional real linear spaces endowed with a bilinear multiplication. It is shown that such cones are semidefinite repre- sentable, and the representation is explicitly constructed. An example of a constraint representable in this form is the pointwise positive semidefiniteness of a time-varying covariance matrix. Finally, seeking a method to circumvent the use of semidefinite programming in the solution of optimization problems over nonnegative and weighted-sum-of-squares polynomials, we study the conditions of optimality (for homotopy methods), as well as barrier functions (for barrier methods) ii in such optimization problems. For univariate polynomials a method based on fast Fourier transform is provided that accelerates the evaluation of the logarithmic barrier function (and its first two derivatives) of the moment cone. This reduces the running time of some interior point methods for semidefinite programming from O(n6) to O(n3) when solving problems involving polynomial nonnegativity constraints. iii Contents Abstract ii 1 Introduction 1 1.1 Outline of the thesis....................................5 2 Nonnegative and sum-of-squares polynomials7 2.1 Univariate nonnegative polynomials...........................9 2.2 Multivariate nonnegative polynomials.......................... 13 2.2.1 Global nonnegativity................................ 13 2.2.2 Nonnegative polynomials on polyhedra...................... 15 2.3 Dual cones of nonnegative polynomials: moment cones................. 18 3 Univariate shape constrained estimation 21 3.1 Existing results in univariate shape-constrained estimation............... 22 3.2 Sieves and cones of nonnegative functions........................ 26 3.2.1 Estimation in functional Hilbert spaces..................... 26 3.2.2 Sieves and finite-dimensional approximations.................. 28 3.3 Representations of nonnegative polynomials and splines................ 32 3.3.1 Representations involving semidefinite and second-order cone constraints.. 32 3.3.2 Sieves of polyhedral spline cones......................... 34 3.4 Shape-constrained statistical estimation models..................... 38 3.4.1 Non-parametric regression of a nonnegative function.............. 39 3.4.2 Isotonic, convex, and concave non-parametric regression............ 39 3.4.3 Unconstrained density estimation......................... 40 3.4.4 Unimodal density estimation........................... 40 iv 3.4.5 Log-concave density estimation.......................... 41 3.4.6 Arrival rate estimation............................... 44 3.4.7 Knot point selection................................ 45 3.5 Numerical examples.................................... 46 3.5.1 Isotonic and convex/concave regression..................... 46 3.5.2 Density estimation................................. 51 3.5.3 Arrival rate estimation............................... 53 3.6 Summary and conclusion................................. 54 4 Multivariate shape constrained estimation 56 4.1 Multivariate nonnegative and sum-of-squares splines.................. 57 4.1.1 Numerical illustration { posterior probability estimation............ 61 4.2 A decomposition method for multivariate splines.................... 63 4.2.1 Augmented Lagrangian decomposition for sparse problems........... 64 4.2.2 Numerical illustration { NJ Turnpike accidents................. 67 4.3 Generalized sum-of-squares cones............................. 70 4.3.1 Semidefinite characterization of sums of squares................. 72 4.3.2 Application to vector-valued functions...................... 78 4.3.3 Further applications................................ 82 4.4 Conclusion......................................... 84 5 Direct optimization methods for nonnegative polynomials 85 5.1 Optimality conditions for nonnegative polynomials................... 87 5.1.1 Preliminary observations.............................. 89 5.1.2 Nonnegative polynomials over a finite interval.................. 91 5.1.3 M¨untz polynomials................................. 100 5.1.4 Cones linearly isomorphic to nonnegative polynomials over an interval.... 102 5.2 A barrier method for moment cone constraints..................... 106 5.2.1 Fast computation of the gradient and the Hessian............... 107 5.2.2 Alternative bases.................................. 109 5.3 Conclusion......................................... 110 6 Conclusion 112 v A Background material 114 A.1 Conic optimization..................................... 114 A.2 Positive semidefinite matrices............................... 116 A.3 Semidefinite and second-order cone programming.................... 117 Bibliography 120 Vita 131 vi 1 Chapter 1 Introduction In this thesis we consider function estimation problems involving constraints on the shape of the estimator or some other underlying function. Such estimation problems abound in statistics and engineering, and there are numerous examples in other disciplines as well. Here we briefly recall a few to give a flavor of these applications; we will return to some of them in the subsequent chapters. 1. Isotonic regression. Regression problems often arise in situations where some prior knowledge (laws of nature or common sense) tells us that the estimated function must be monotone increasing or decreasing in some of the variables. Calculating the least-squares regression function under this constraint is the simplest form of the univariate isotonic regression problem [RWD88]. Multivariate extensions, as well as more abstract forms of this problem, such as [Bru55], also exist; in these problems monotonicity is understood with respect to some partial order. 2. Convex/Concave regression. Similarly to isotonic regression, some problems may require that the estimator is convex or concave over its domain [Hil54]. Univariate applications include the estimation of the Laffer curve [Wan78] and utility functions in econometrics. The latter is also an example of a situation when multiple shape constraints may be present: the utility function of a rational, risk-averse decision maker is increasing and concave. 3. Density estimation. One of the fundamental problems in statistics is the estimation of probability density functions (or PDFs) and cumulative distribution functions (or CDFs) of random variables, given a finite number of samples (realizations) of these variables [DG85, EL01]. These problems come in two flavors: in the parametric setting the unknown PDF or CDF is 2 assumed to belong to a finite-dimensional space (or at least to a family of functions given up to a finite number of parameters) that contains only PDFs and CDFs. Usually one considers parametric families which contain only functions of the right shape. In the non-parametric setting, however, no such assumptions are made, and constraints such as nonnegativity, being less than or equal to one, and monotonicity must be explicitly added to the problem. Further constraints, such as the unimodality or log-concavity of the PDF can also be considered (even though unimodality is not a convex constraint, which makes optimization with unimodality constraints particularly difficult). 4. Arrival rate estimation. The arrival rate of a non-homogeneous