5 - 1 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 5. Sum of Squares Polynomial nonnegativity ² Sum of squares (SOS) decompositions ² Computing SOS using semide¯nite programming ² Liftings ² Dual side: moments ² Applications ² Global optimization ² Optimizing in parameter space ² Lyapunov functions ² Density functions and control synthesis ² 5 - 2 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Polynomial Nonnegativity Before dealing with systems of polynomial inequalities, we study the sim- plest nontrivial problem: one inequality. Given f(x1; : : : ; xn) (of even degree), is it globally nonnegative? n f(x1; x2; : : : ; xn) 0; x R ¸ 8 2 For quadratic polynomials (d = 2), very easy. Essentially, checking if ² a matrix is PSD. The problem is NP-hard when d 4. ² ¸ Problem is decidable, algorithms exist (more later). Very powerful, ² but bad complexity properties. Many applications. We'll see a few. ² 5 - 3 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Sum of Squares Decomposition A \simple" su±cient condition: a sum of squares (SOS) decomposition: 2 f(x) = g (x); gi R[x] i 2 Xi If f(x) can be written as above, then f(x) 0. ¸ A purely syntactic, easily veri¯able certi¯cate. Always a su±cient condition for nonnegativity. In some cases (univariate, quadratic, etc.), also necessary. But in general, SOS is not equivalent to nonnegativity. However, a very good thing: we can compute this e±ciently using SDP. 5 - 4 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Sum of Squares and SDP Consider a polynomial f(x1; : : : ; xn) of degree 2d. Let z be a vector with all monomials of degree less than or equal to d. n+d The number of components of z is d . Then, f is SOS i®: ¡ ¢ f(x) = zT Qz; Q 0 º Factorize Q = LT L. Then ² f(x) = zT LT Lz = Lz 2 = (Lz)2 jj jj i Xi The terms in the SOS decomposition are given by g = (Lz) . ² i i The number of squares is equal to the rank of Q. ² 5 - 5 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 f(x) = zT Qz; Q 0 º Comparing terms, we obtain linear equations for the elements of Q. ² The desired matrices Q lie in the intersection of an a±ne set of ma- ² trices, and the PSD cone. In general, Q is not unique. ² Can be solved as semide¯nite program in the standard primal form. ² Q 0; trace A Q = b f º i ig 5 - 6 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Multivariate SOS Example f(x; y) = 2x4 + 5y4 x2y2 + 2x3y ¡ 2 T 2 x q11 q12 q13 x = y2 q q q y2 2 3 2 12 22 23 3 2 3 xy q13 q23 q33 xy 4 45 4 4 5 4 25 2 3 3 = q11x + q22y + (q33 + 2q12)x y + 2q13x y + 2q23xy The existence of a PSD Q is exactly equivalent to feasibility of an SDP in the standard primal form: q11 = 2 q22 = 5 Q 0; subject to 2q = 0 2q = 2 º 8 23 13 <> q + 2q = 1 33 12 ¡ :> 5 - 7 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Multivariate SOS Example (continued) Solving numerically, we obtain a particular solution: 2 3 1 ¡ 1 2 3 1 Q = 3 5 0 = LT L; L = ¡ 2 ¡ 3 p2 0 1 3 1 0 5 · ¸ 4 5 This Q has rank two, therefore f(x; y) is the sum of two squares: 1 1 f(x; y) = (2x2 3y2 + xy)2 + (y2 + 3xy)2 2 ¡ 2 This representation certi¯es nonnegativity of f. Using SOSTOOLS: [Q,Z]=findsos(2*x^4+5*y^4-x^2*y^2+2*x^3*y) 5 - 8 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Some Background In 1888, Hilbert showed that PSD=SOS if and only if ² d = 2. Quadratic polynomials. SOS decomposition follows from ² Cholesky, square root, or eigenvalue decomposition. n = 1. Univariate polynomials. ² d = 4; n = 2. Quartic polynomials in two variables. ² Connections with Hilbert's 17th problem, solved by Artin: every PSD ² polynomial is a SOS of rational functions. If f is not SOS, then can try with gf, for some g. ² For ¯xed f, can optimize over g too ² Otherwise, can use a \universal" construction of P¶olya-Reznick. ² More about this later. 5 - 9 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 M(x,y,1) The Motzkin Polynomial A positive semide¯nite polynomial, 1.2 that is not a sum of squares. 1 0.8 2 4 4 2 2 2 0.6 M(x; y) = x y + x y + 1 3x y 0.4 ¡ 0.2 0 1 0.5 1 0 0.5 0 −0.5 −0.5 −1 −1 y x Nonnegativity follows from the arithmetic-geometric inequality ² applied to (x2y4; x4y2; 1) Introduce a nonnegative factor x2 + y2 + 1 ² Solving the SDPs we obtain the decomposition: ² (x2 + y2 + 1) M(x; y) = (x2y y)2 + (xy2 x)2 + (x2y2 1)2+ ¡ ¡ ¡ 1 3 + (xy3 x3y)2 + (xy3 + x3y 2xy)2 4 ¡ 4 ¡ 5 - 10 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 The Univariate Case: 2 3 2d f(x) = a0 + a1x + a2x + a3x + + a2dx T ¢ ¢ ¢ 1 q00 q01 : : : q0d 1 x q q : : : q x = 01 11 1d 2 . 3 2 . ... 3 2 . 3 6 xd 7 6 q q : : : q 7 6 xd 7 6 7 6 00 1d dd 7 6 7 4d 5 4 5 4 5 i = qjk x Xi=0j+Xk=i ¶ In the univariate case, the SOS condition is exactly equivalent to non- ² negativity. The matrices Ai in the SDP have a Hankel structure. This can be ² exploited for e±cient computation. 5 - 11 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 A General Method: Liftings Consider this polytope in R3 (a zonotope). It has 56 facets, and 58 vertices. Optimizing a linear function over this set, re- quires a linear program with 56 constraints (one per face). However, this polyhedron is a three- dimensional projection of the 8-dimensional 8 hypercube x R ; 1 xi 1 . f 2 ¡ · · g Therefore, by using additional variables, we can solve the same problem, by using an LP with only 16 constraints. 5 - 12 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Liftings By going to higher dimensional representations, things may become easier: \Complicated" sets can be the projection of much simpler ones. ² A polyhedron in Rn with a \small" number of faces can project to a ² lower dimensional space with exponentially many faces. Basic semialgebraic sets can project into non-basic semialgebraic sets. ² An essential technique in integer programming. Advantages: compact representations, avoiding \case distinctions," etc. 5 - 13 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 Example minimize (x 3)2 ¡ subject to x (x 4) 0 ¡ ¸ The feasible set is [ ; 0] [4; ]. Not convex, or even connected. ¡1 [ 1 Consider the lifting L : R R2, with L(x) = (x; x2) =: (x; y). ! Rewrite the problem in terms of the lifted variables. y 1 x 30 For every lifted point, 0. ² x y º · ¸ 20 Constraint becomes: y 4x 0 ² ¡ ¸ 10 Objective is now: y 6x + 9 ² ¡ -2 2 4 6 x We \get around" nonconvexity: interior points are now on the boundary. 5 - 14 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 The Dual Side of SOS: Moment Sequences The SDP dual of the SOS construction gives e±cient semide¯nite liftings. For the univariate case: L : R Sd+1, with ! 1 x : : : xd x x2 : : : xd+1 L(x) = 2 3 . ... 6 d d+1 2d 7 6 x x : : : x 7 6 7 4 5 The matrices L(x) are Hankel, positive semide¯nite, and rank one. The convex hull co L(x) therefore contains only PSD Hankel matrices. 1 w1 : : : wd w w : : : w Hankel(w) := 1 2 d+1 2 . ... 3 6 w w : : : w 7 6 d d+1 2d 7 4 5 (in fact, in the univariate case every PSD Hankel is in the convex hull) 5 - 15 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 SOS Dual (continued) For nonnegativity, want to rule out the existence of x with f(x) < 0. In the lifted variables, we can look at: Hankel(w) 0; a w < 0 º i i ½ Xi This is exactly the SDP dual of the univariate SOS construction. Q 0; q = a º jk i ½ j+Xk=i If the ¯rst problem is feasible, there is always a w such that Hankel(w) ² is rank one. It corresponds directly to the lifting of a primal point. Direct extensions to the multivariate case. Though in general, PSD = SOS. 6 5 - 16 Sum of Squares P. Parrilo and S. Lall, ECC 2003 2003.09.02.03 A General Scheme Nonnegativity Relaxation Lifting and convex hull SDP Lifted problem Duality Sum of squares Lifting corresponds to a classical problem of moments. ² The solution to the lifted problem may suggest candidate points where ² the polynomial is negative. The sums of squares certify or prove polynomial nonnegativity.