The ellipsoid method We have learned that the Markowitz mean-variance optimization problem is a convex programming problem. The good news is that there are effi- cient methods to solve such convex programming problems and in particu- lar quadratic programming problems in practice. In this lecture we briefly sketch a theoretical result, which yields an approximate polynomial time algorithm for convex programming problems. Our goal is to solve a convex optimization problem min f0(x) fi(x) 6 bi, i = 1,...,m. We reduce the optimization problem to the decision problem as follows. We ∗ ∗ are trying to find a β R such that f0(x) 6 β ,f1(x) 6 b1,...,fm(x) 6 bm ∈ ∗ is feasible, while f0(x) 6 β ε, f1(x) 6 b1,...,fm(x) 6 bm is infeasible. − ∗ ∗ Suppose we know numbers L 6 β ε and U > β . We now test whether − f0(x) 6 (L + U)/2,f1(x) 6 b1,...,fm(x) 6 bm is feasible. If yes, we set U = (L+U)/2 and if no, we set L = (L+U)/2. After O(log(U L) log(ε)) − − many steps, this procedure terminates. This approximates the optimum value of the convex program. This leaves us with the problem to decide whether a convex body is nonempty or not. Let K Rn be a compact convex set with volume vol(K). ⊆ ∗ Initially, the ellipsoid method can be used to determine a point x K or to ∈ assert that the volume of K is less than a certain lower bound L. The unit ball is the set B = x Rn x 6 1 and an ellipsoid E(A, b) is { ∈ | k k } the image of the unit ball under a linear map t : Rn Rn with t(x)= Ax+b, × → where A Rn n is an invertible matrix and b Rn is a vector. Clearly ∈ ∈ − − E(A, b)= x Rn A 1x A 1b 6 1 . (1.1) { ∈ | k − k } 1 3 x(1) Exercise 1. Consider the mapping t(x) = 2 5 x(2) . Draw the ellipsoid t which is defined by . What are the axes of the ellipsoid? n/2 The volume of the unit ball is denoted by V , where V 1 2 eπ . n n ∼ πn n It follows that the volume of the ellipsoid E(A, b) is equal to det(A) V . | | · n The next lemma is the key to the development of the ellipsoid method. 1 Lemma 1 (Half-Ball Lemma). The half-ball H = x Rn x 6 1,x(1) > { ∈ | k k 0 is contained in the ellipsoid } n n + 1 2 1 2 n2 1 Rn 2 6 E = x x(1) + −2 x(i) 1 (1.2) ( ∈ | n − n + 1 n ) Xi=2 x(1) > 0 Figure 1.1: Half-ball lemma. Proof. Let x be contained in the unit ball, i.e., x 6 1 and suppose further k k that 0 6 x(1) holds. We need to show that n n + 1 2 1 2 n2 1 x(1) + − x(i)2 6 1 (1.3) n − n + 1 n2 Xi=2 holds. Since n x(i)2 6 1 x(1)2 holds we have i=2 − n n + 1 2 P 1 2 n2 1 x(1) + − x(i)2 n − n + 1 n2 Xi=2 n + 1 2 1 2 n2 1 6 x(1) + − (1 x(1)2) n − n + 1 n2 − (1.4) This shows that (1.3) holds if x is contained in the half-ball and x(1) = 0 or x(1) = 1. Now consider the right-hand-side of (1.4) as a function of x(1), i.e., consider n + 1 2 1 2 n2 1 f(x(1)) = x(1) + − (1 x(1)2). (1.5) n − n + 1 n2 − 2 The first derivative is 2 2 ′ n + 1 1 n 1 f (x(1)) = 2 x(1) 2 − x(1). (1.6) · n − n + 1 − · n2 ′ We have f (0) < 0 and since both f(0) = 1 and f(1) = 1, we have f(x(1)) 6 1 for all 0 6 x(1) 6 1 and the assertion follows. In terms of a matrix A and a vector b, the ellipsoid E is described as − − E = x Rn A 1x A 1b 6 1 , where A is the diagonal matrix with { ∈ | k − k } diagonal entries n n2 n2 , ,..., n + 1 n2 1 n2 1 r − r − and b is the vector b = (1/(n + 1), 0,..., 0). Our ellipsoid E is thus the image of the unit sphere under the linear transformation t(x)= Ax+b. The (n−1)/2 n n2 6 x determinant of A is thus n+1 n2−1 . Using the inequality 1+x e we see that this is bounded by − − · 2 − − 1 e 1/(n+1)e(n 1)/(2 (n 1)) = e 2(n+1) . (1.7) We can conclude the following theorem. Theorem 2. The half-ball x Rn x(1) > 0, x 6 1 is contained in an { ∈ | − 1 k k } ellipsoid E, whose volume is bounded by e 2(n+1) V . · n Recall the following notion from linear algebra. A symmetric matrix × A Rn n is called positive definite if all its eigenvalues are positive. Recall ∈ the following theorem. Theorem 3. Let A Rn×n be a symmetric matrix. The following are equivalent. ∈ i) A is positive definite. ii) A = LT L, where L Rn×n is a uniquely determined upper triangular ma- ∈ trix. iii) xT Ax > 0 for each x Rn 0 . ∈ \ { } iv) A = QT diag(λ ,...,λ )Q, where Q Rn×n is an orthogonal matrix and 1 n ∈ λ R for i = 1,...,n. i ∈ >0 It is now convenient to switch to a different representation of an ellip- − soid. An ellipsoid E (A, a) is the set E (A, a) = x Rn (x a)T A 1(x × { ∈ | − − a) 6 1 , where A Rn n is a symmetric positive definite matrix and a Rn } ∈ ∈ is a vector. Consider the half-ellipsoid E (A, a) (cT x 6 cT a). ∩ Our goal is a similar lemma as the half-ball-lemma for ellipsoids. Geo- metrically it is clear that each half-ellipsoid E (A, a) (cT x 6 cT a) must be ∩ 3 ′ ′ ′ ′ contained in another ellipsoid E (A , b ) with vol(E (A , a ))/vol(E (A, a)) 6 − e 1/(2n). More precisely this follows from the fact that the half-ellipsoid is the image of the half-ball under a linear transformation. Therefore the image of the ellipsoid E under the same transformation contains the half- ellipsoid. Also, the volume-ratio of the two ellipsoids is invariant under a linear transformation. ′ ′ ′ We now record the formula for the ellipsoid E (A , a ). It is defined by ′ 1 a = a b (1.8) − n + 1 2 ′ n 2 A = A b bT , (1.9) n2 1 − n + 1 − where b is the vector b = A c/√cT A c. The proof of the correctness of this formula can be found in [1]. Lemma 4 (Half-Ellipsoid-Theorem). The half-ellipsoid E (A, b) (cT x 6 cT a) ′ ′ ′ ′ ∩ − is contained in the ellipsoid E (A , a ) and one has vol(E )/vol(E ) 6 e 1/(2n). The method Suppose we know an ellipsoid Einit which contains K. The ellipsoid method is described as follows. The input to the ellipsoid method is Einit and a pos- itive number L. The mothod either i) asserts that vol(K) <L or ∗ ii) finds a point x K. ∈ Algorithm (Ellipsoid method exact version). a) (Initialize): Set E (A, a) := Einit b) If a K, then assert K = and stop ∈ 6 ∅ c) If vol(E ) <L, then assert that vol(K) <L. d) Otherwise, compute an inequality cT x 6 β which is valid for K and ′ satisfies cT a > β and replace E (A, a) by E (A , a) computed with for- mula (1.8) and goto step b). Theorem 5. The ellipsoid method computes a point in K or asserts that vol(K) < L. The number of iterations is bounded by 2 n ln(vol(E )/L). · init Proof. After i iterations one has − i vol(E )/vol(Einit) 6 e 2n . (1.10) Since we stop when vol(E ) < L, we stop at least after 2 n ln(vol(E )/L) · init iterations. This shows the claim. 4 1.1 Linear programming In this section we discuss how the ellipsoid method can solve a linear pro- gram in polynomial time. This result was shown by Khachiyan [2] in 1979 and solved a longstanding open problem at that time. Feasibility versus optimization We firstshow that it is enoughto have an algorithm for the feasibility problem of linear inequalities. × Given A Zm n and b Zm defining a polyhedron P = x ∈ ∈ ∗ { ∈ Rn : Ax 6 b , compute a point x P or assert that P is empty. } ∈ That this is enough is easily seen by linear programming duality. A ∗ point x Rn is an optimal solution of the linear program max cT x: x ∈ ∗ ∗ { ∈ Rn Rm x∗ , Ax 6 b if and only if there exists a y such that y is contained } x n+m T T∈ T in the polyhedron y R : c x = b y, Ax 6 b, A y = c, y > 0 . { ∈ } Our goal is therefore to show that the ellipsoid method can solve the feasibility problem in a polynomial number of iterations. More precisely, we are showing the following theorem.