Linear Programming 1 an Introduction to Linear Programming

18.415/6.854 Advanced Algorithms October 1994 Linear Programming Lecturer: Michel X. Goemans

1 An Introduction to Linear Programming

Linear programming is a very important class of problems, both algorithmically and combinatorially. Linear programming has many applications. From an algorithmic point-of-view, the simplex was proposed in the forties (soon after the war, and was motivated by military applications) and, although it has performed very well in practice, is known to run in exponential time in the worst-case. On the other hand, since the early seventies when the classes P and NP were defined, it was observed that linear programming is in NP co-NP although no polynomial-time algorithm was known at ∩ that time. The first polynomial-time algorithm, the ellipsoid algorithm, was only discovered at the end of the seventies. Karmarkar’s algorithm in the mid-eighties lead to very active research in the area of interior-point methods for linear programming. We shall present one of the numerous variations of interior-point methods in class. From a combinatorial perspective, systems of linear inequalities were already studied at the end of the last century by Farkas and Minkovsky. Linear programming, and especially the notion of duality, is very important as a proof technique. We shall illustrate its power when discussing approximation algorithms. We shall also talk about network flow algorithms where linear programming plays a crucial role both algorithmically and combinatorially. For a more in-depth coverage of linear programming, we refer the reader to [1, 4, 7, 8, 5]. A linear program is the problem of optimizing a linear objective function in the decision variables, x1 ...xn, subject to linear equality or inequality constraints on the xi’s. In standard form, it is expressed as: n Min cjxj (objective function) j=1 subject to: n aijxj = bi, i =1 ...m (constraints) j=1 x 0, j =1 ...n (non-negativity constraints) j ≥ where a , b ,c are given. { ij i j} A linear program is expressed more conveniently using matrices: Ax = b min cT x subject to x 0 ≥ LP-1 where x1 . Rn 1 x =  .  × ∈  x   n   b1  . Rm 1 b =  .  × ∈  b   m   c1  . Rn 1 c =  .  × ∈  c   n   a11  . m n A =  ..  R × ∈  a   mn    2 Basic Terminology

Definition 1 If x satisfies Ax = b, x 0, then x is feasible. ≥ Definition 2 A linear program (LP) is feasible if there exists a feasible solution, otherwise it is said to be infeasible. T T Definition 3 An optimal solution x∗ is a feasible solution s.t. c x∗ = min c x : Ax = b, x 0 . { ≥ } T Definition 4 LP is unbounded (from below) if λ R, a feasible x∗ s.t. c x∗ λ. ∀ ∈ ∃ ≤ 3 Equivalent Forms

A linear program can take on several forms. We might be maximizing instead of minimizing. We might have a combination of equality and inequality contraints. Some variables may be restricted to be non-positive instead of non-negative, or be unrestricted in sign. Two forms are said to be equivalent if they have the same set of optimal solutions or are both infeasible or both unbounded. 1. A maximization problem can be expressed as a minimization problem. max cT x min cT x ⇔ − 2. An equality can be represented as a pair of inequalities. T ai x bi T ≤ T ⇔ ai x bi ai x = bi T ≥ ai x bi T ≤ ⇔ a x bi − i ≤ −

LP-2 3. By adding a slack variable, an inequality can be represented as a combination of equality and non-negativity constraints.

aT x b aT x + s = b , s 0. i ≤ i ⇔ i i i i ≥

4. Non-positivity constraints can be expressed as non-negativity constraints.

To express xj 0, replace xj everywhere with yj and impose the condition y 0. ≤ − j ≥ 5. x may be unrestricted in sign. If x is unrestricted in sign, i.e. non-positive or non-negative, everywhre replace + + x by x x−, adding the constraints x , x− 0. j j − j j j ≥

In general, an inequality can be represented using a combination of equality and non-negativity constraints, and vice versa. T T + T Using these rules, min c x s.t. Ax b can be transformed into min c x c x− + + ≥ } − s.t. Ax Ax− Is = b,x , x−,s 0 . The former LP is said to be in canonical − − ≥ } form, the latter in standard form. Conversely, an LP in standard form may be written in canonical form. min cT x s.t. Ax = b, x 0 is equivalent to min cT x s.t. Ax b, Ax b, Ix 0 . ≥ } ≥ − ≥ − ≥ } A b ′ ′ ′ ′ This may be rewritten as A x b , where A = -A and b = -b . ≥     I 0         4 Example

Consider the following linear program:

x 2 1 ≥  3x1 x2 0 min x2 subject to − ≥  x1 + x2 6  ≥ x + 2x 0 − 1 2 ≥   The optimal solution is (4, 2) of cost 2 (see Figure 1). If we were maximizing x2 instead of minimizing under the same feasible region, the resulting linear program would be unbounded since x2 can increase arbitrarily. From this picture, the reader should be convinced that, for any objective function for which the linear program is bounded, there exists an optimal solution which is a “corner” of the feasible region. We shall formalize this notion in the next section.

LP-3 3x − x21 > 0

(2,4) −x1 + 2x2 > 0

x 1+ x2 > 6 (2,1)

x1 > 2

Figure 1: Graph representing primal in example.

An example of an infeasible linear program can be obtained by reversing some of the inequalities of the above LP:

x1 2 3x x ≤ 0 1 − 2 ≥ x1 + x2 6 x + 2x ≥ 0. − 1 2 ≤ 5 The Geometry of LP

Let P = x : Ax = b, x 0 Rn. { ≥ } ⊆ Definition 5 x is a vertex of P if y =0 s.t. x + y, x y P . ∃ − ∈ ′ Theorem 1 Assume min cT x : x P is finite, then x P, a vertex x such that ′ { ∈ } ∀ ∈ ∃ cT x cT x. ≤ Proof: ′ If x is a vertex, then take x = x. If x is not a vertex, then, by definition, y = 0 s.t. x + y, x y P . Since A(x + y)= b and A(x y)= b, Ay = 0. ∃ − ∈ − LP-4 WLOG, assume cT y 0 (take either y or y). If cT y = 0, choose y such that j s.t. y < 0. Since y = 0 and≤ cT y = cT ( y) = 0,− this must be true for either y or ∃y. j − − Consider x + λy, λ > 0. cT (x + λy) = cT x + λcT y cT x, since cT y is assumed ≤ non-positive.

Case 1 j such that y < 0 ∃ j As λ increases, component j decreases until x + λy is no longer feasible.

Choose λ = min j:yj<0 xj/ yj = xk/ yk. This is the largest λ such that { } x + λy 0. Since Ay ={ 0, A−(x +} λy)= −Ax + λAy = Ax = b. So x + λy P , ≥ ∈ and moreover x + λy has one more zero component, (x + λy)k, than x. Replace x by x + λy.

Case 2 y 0 j j ≥ ∀ By assumption, cT y < 0 and x + λy is feasible for all λ 0, since A(x + λy)= Ax + λAy = Ax = b, and x + λy x 0. But cT (x + λy≥)= cT x + λcT y ≥ ≥ → −∞ as λ , implying LP is unbounded, a contradiction. →∞ Case 1 can happen at most n times, since x has n components. By induction on ′ the number of non-zero components of x, we obtain a vertex x . 2 Remark: The theorem was described in terms of the polyhedral set P = x : Ax = b : x 0 . Strictly speaking, the theorem is not true for P = x : Ax{ b . Indeed, such≥ } a set P might not have any vertex. For example, consider{ P ≥= } (x1, x2):0 x2 1 (see Figure 2). This polyhedron has no vertex, since for any x{ P , we have≤ x≤+ y}, x y P , where y = (1, 0). It can be shown that P has a ∈ − ∈ vertex iﬀ Rank(A)= n. Note that, if we transform a program in canonical form into standard form, the non-negativity constraints imply that the resulting matrix A has full column rank, since A Rank  -A  = n. I    

Corollary 2 If min cT x : Ax = b, x 0 is ﬁnite, There exists an optimal solution, { ≥ } x∗, which is a vertex.

Proof: Suppose not. Take an optimal solution. By Theorem 1 there exists a vertex costing no more and this vertex must be optimal as well. 2

Corollary 3 If P = x : Ax = b, x 0 = , then P has a vertex. { ≥ } ∅

LP-5 x2

(0,1)

(0,0) x1

Figure 2: A polyhedron with no vertex.

Theorem 4 Let P = x : Ax = b, x 0 . For x P , let A be a submatrix of A { ≥ } ∈ x corresponding to j s.t. xj > 0. Then x is a vertex iﬀ Ax has linearly independent columns. (i.e. Ax has full column rank.) 2 2130 2 3 0 Example A = 7321 x =   A = 7 2 , and x is a vertex.   1 x   0005   0 0    0          Proof:   Show i ii. ¬ → ¬ Assume x is not a vertex. Then, by deﬁnition, y = 0 s.t. x + y, x y P . ∃ − ∈ Let Ay be submatrix corresponding to non-zero components of y. As in the proof of Theorem 1, Ax + Ay = b Ay =0. Ax Ay = b ⇒ − Therefore, A has dependent columns since y = 0. y Moreover, x + y 0 ≥ yj = 0 whenever xj =0. x y 0 ⇒ − ≥

Therefore Ay is a submatrix of Ax. Since Ay is a submatrix of Ax, Ax has linearly dependent columns.

LP-6 Show ii i. ¬ → ¬ Suppose Ax has linearly dependent columns. Then y s.t. Axy = 0, y = 0. Extend y to Rn by adding 0 components. Then y ∃ Rn s.t. Ay = 0, y = 0 ∃ ∈ and yj = 0 wherever xj = 0. ′ ′ ′ Consider y = λy for small λ 0. Claim that x + y , x y P , by argument analogous to that in Case 1 of≥ the proof of Theorem 1,− above.∈ Hence, x is not a vertex. 2

6 Bases

Let x be a vertex of P = x : Ax = b, x 0 . Suppose ﬁrst that j : x > 0 = m { ≥ } |{ j }| (where A is m n). In this case we denote B = j : xj > 0 . Also let AB = Ax; we use this notation× not only for A and B, but also{ for x and} for other sets of indices. Then AB is a square matrix whose columns are linearly independent (by Theorem 4), so it is non-singular. Therefore we can express x as xj = 0 if j B, and since 1 ∈ ABxB = b, it follows that xB = AB− b. The variables corresponding to B will be called basic. The others will be referred to as nonbasic. The set of indices corresponding to nonbasic variables is denoted by N = 1,...,n B. Thus, we can write the above 1 { } − as xB = AB− b and xN = 0. Without loss of generality we will assume that A has full row rank, rank(A)= m. Otherwise either there is a redundant constraint in the system Ax = b (and we can remove it), or the system has no solution at all. If j : x > 0 < m, we can augment A with additional linearly independent |{ j }| x columns, until it is an m m submatrix of A of full rank, which we will denote AB. In other words, although× there may be less than m positive components in x, it is convenient to always have a basis B such that B = m and AB is non-singular. This | | 1 enables us to always express x as we did before, xN = 0, xB = AB− b.

Summary x is a vertex of P iﬀ there is B 1,...,n such that B = m and ⊆{ } | | 1. x = 0 for N = 1,...,n B N { } −

2. AB is non-singular

1 3. x = A− b 0 B B ≥ In this case we say that x is a basic feasible solution. Note that a vertex can have several basic feasible solution corresponding to it (by augmenting j : xj > 0 in { 1} diﬀerent ways). A basis might not lead to any basic feasible solution since AB− b is not necessarily nonnegative.

LP-7 Example:

x1 + x2 + x3 =5 2x1 x2 +2x3 =1 x , x−, x 0 1 2 3 ≥ We can select as a basis B = 1, 2 . Thus, N = 3 and { } { } 1 1 A = B 2 1 − 1 1 1 3 3 AB− = 2 1 3 −3

1 2 AB− b = 3 x = (2, 3, 0)

n Remark. A crude upper bound on the number of vertices of P is m . This number is exponential (it is upper bounded by nm). We can come up with a tighter approx- m n 2 imation of −m , though this is still exponential. The reason why the number is 2 much smaller is that most basic solutions to the system Ax = b (which we counted) are not feasible, that is, they do not satisfy x 0. ≥

7 The Simplex Method

The Simplex algorithm [Dantzig,1947] [2] solves linear programming problems by focusing on basic feasible solutions. The basic idea is to start from some vertex v and look at the adjacent vertices. If an improvement in cost is possible by moving to one of the adjacent vertices, then we do so. Thus, we will start with a bfs corresponding to a basis B and, at each iteration, try to improve the cost of the solution by removing one variable from the basis and replacing it by another. We begin the Simplex algorithm by ﬁrst rewriting our LP in the form:

min cBxB + cN xN s.t. ABxB + AN xN = b x , x 0 B N ≥ Here B is the basis corresponding to the bfs we are starting from. Note that, for 1 1 T any solution x, xB = AB− b AB− AN xN and that its total cost, c x can be speciﬁed as follows: −

LP-8 T c x = cBxB + cN xN 1 1 = cB(AB− b AB− AN xN )+ cN xN 1 − 1 = c A− b +(c c A− A )x B B N − B B N N 1 We denote the reduced cost of the non-basic variables byc Ñ ,c Ñ = cN cBAB− AN , i.e. the quantity which is the coefficient of x above. If there is a j −N such that N ∈ c˜j < 0, then by increasing xj (up from zero) we will decrease the cost (the value of the objective function). Of course xB depends on xN , and we can increase xj only as long as all the components of xB remain positive. So in a step of the Simplex method, we find a j N such thatc ˜j < 0, and increase it as much as possible while keeping x 0. It is∈ not possible any more to increase B ≥ xj, when one of the components of xB is zero. What happened is that a non-basic variable is now positive and we include it in the basis, and one variable which was basic is now zero, so we remove it from the basis. If, on the other hand, there is no j N such thatc ˜j < 0, then we stop, and the current basic feasible solution is an optimal∈ solution. This follows from the new T expression for c x since xN is nonnegative.

Remarks:

1. Note that some of the basic variables may be zero to begin with, and in this case it is possible that we cannot increase xj at all. In this case we can replace say j by k in the basis, but without moving from the vertex corresponding to the basis. In the next step we might replace k by j, and be stuck in a loop. Thus, we need to specify a “pivoting rule” to determine which index should enter the basis, and which index should be removed from the basis.

2. While many pivoting rules (including those that are used in practice) can lead to inﬁnite loops, there is a pivoting rule which will not (known as the minimal index rule - choose the minimal j and k possible [Bland, 1977]). This fact was discovered by Bland in 1977. There are other methods of “breaking ties” which eliminate inﬁnite loops.

3. There is no known pivoting rule for which the number of pivots in the worst case is better than exponential.

4. The question of the complexity of the Simplex algorithm and the last remark leads to the question of what is the length of the shortest path between two vertices of a convex polyhedron, where the path is along edges, and the length of the path in measured in terms of the number of vertices visited.

LP-9 Hirsch Conjecture: For m hyperplanes in d dimensions the length of the shortest path between any two vertices of the arrangement is at most m d. − This is a very open question — there is not even a polynomial bound proven on this length. On the other hand, one should note that even if the Hirsch Conjecture is true, it doesn’t say much about the Simplex Algorithm, because Simplex generates paths which are monotone with respect to the objective function, whereas the shortest path need not be monotone. Recently, Kalai (and others) has considered a randomized pivoting rule. The idea is to randomly permute the index columns of A and to apply the Simplex method, always choosing the smallest j possible. In this way, it is possible to show a subexponential bound on the expected number of pivots. This leads to a subexponential bound for the diameter of any convex polytope defined by m hyperplanes in a d dimension space. The question of the existence of a polynomial pivoting scheme is still open though. We will see later a completely different algorithm which is polynomial, although not strongly polynomial (the existence of a strongly polynomial algorithm for linear programming is also open). That algorithm will not move from one vertex of the feasible domain to another like the Simplex, but will confine its interest to points in the interior of the feasible domain.

A visualization of the geometry of the Simplex algorithm can be obtained from considering the algorithm in 3 dimensions (see Figure 3). For a problem in the form min cT x : Ax b the feasible domain is a polyhedron in R3, and the algorithm { ≤ } moves from vertex to vertex in each step (or does not move at all).

8 When is a Linear Program Feasible ?

We now turn to another question which will lead us to important properties of linear programming. Let us begin with some examples. We consider linear programs of the form Ax = b, x 0. As the objective function has no eﬀect on the feasibility of the program, we ignore≥ it. We ﬁrst restrict our attention to systems of equations (i.e. we neglect the non- negativity constraints).

Example: Consider the system of equations: x1 + x2 + x3 = 6 2x1 + 3x2 + x3 = 8 2x1 + x2 + 3x3 = 0 and the linear combination

LP-10 Objective function

Figure 3: Traversing the vertices of a convex body (here a polyhedron in R3).

4 x1 + x2 + x3 = 6 −1 × 2x + 3x + x = 8 × 1 2 3 1 2x1 + x2 + 3x3 = 0 The linear× combination results in the equation

0x +0x +0x = 16 1 2 3 − which means of course that the system of equations has no feasible solution. In fact, an elementary theorem of linear algebra says that if a system has no solution, there is always a vector y such as in our example (y = ( 4, 1, 1)) which proves that the system has no solution. −

Theorem 5 Exactly one of the following is true for the system Ax = b: 1. There is x such that Ax = b.

2. There is y such that AT y =0 but yT b =1.

This is not quite enough for our purposes, because a system can be feasible, but still have no non-negative solutions x 0. Fortunately, the following lemma ≥ establishes the equivalent results for our system Ax = b, x 0. ≥ Theorem 6 (Farkas’ Lemma) Exactly one of the following is true for the system Ax = b, x 0: ≥ LP-11 1. There is x such that Ax = b, x 0. ≥ 2. There is y such that AT y 0 but bT y < 0. ≥ Proof: We will ﬁrst show that the two conditions cannot happen together, and then than at least one of them must happen. Suppose we do have both x and y as in the statement of the theorem.

Ax = b = yT Ax = yT b = xT AT y = yT b ⇒ ⇒ but this is a contradiction, because yT b < 0, and since x 0 and AT y 0, so xT AT y 0. ≥ ≥ ≥ The other direction is less trivial, and usually shown using properties of the Sim- plex algorithm, mainly duality. We will use another tool, and later use Farkas’ Lemma to prove properties about duality in linear programming. The tool we shall use is the Projection theorem, which we state without proof:

Theorem 7 (Projection Theorem) Let K be a closed convex (see Figure 4) non- empty set in Rn, and let b be any point in Rn. The projection of b onto K is a point p K that minimizes the Euclidean distance b p . Then p has the property that for∈ all z K, (z p)T (b p) 0 (see Figure 5)− non-empty set. ∈ − − ≤

not convex convex

Figure 4: Convex and non-convex sets in R2.

We are now ready to prove the other direction of Farkas’ Lemma. Assume that there is no x such that Ax = b, x 0; we will show that there is y such that AT y 0 ≥ ≥ but yT b< 0. Let K = Ax : x 0 Rm (A is an m n matrix). K is a cone in Rm and it is { ≥ } ⊆ × convex, non-empty and closed. According to our assumption, Ax = b, x 0 has no solution, so b does not belong to K. Let p be the projection of b onto K.≥

LP-12 z

b p

Figure 5: The Projection Theorem.

Since p K, there is a w 0 such that Aw = p. According to the Projection Theorem, for∈ all z K,(z p)T≥(b p) 0 That is, for all x 0(Ax p)T (b p) 0 ∈ − − ≤ ≥ − − ≤ We deﬁne y = p b, which implies (Ax p)T y 0. Since Aw = p,(Ax Aw)T y − − ≥ − ≥ 0. (x w)T (AT y) 0 for all x 0 (remember that w was ﬁxed by choosing b). − ≥ ≥ 0  0  .  .  Set x = w +   (w plus a unit vector with a 1 in the i-th row). Note that x    1     .   .     0    is non-negative, because  w 0. ≥ This will extract the i-th column of A, so we conclude that the i-th component of AT y is non-negative (AT y) 0, and since this is true for all i, AT y 0. i ≥ ≥ Now it only remains to show that yT b< 0. ytb =(p y)T y = pT y yT y Since (Ax p)T y 0 for all x 0, taking x to be zero shows that p−T y 0. Since−b K, y = p −b = 0, so≥ yT y > 0. So≥ yT b = pT y yT y < 0. ≤ ∈ − − 2 Using a very similar proof one can show the same for the canonical form:

Theorem 8 Exactly one of the following is true for the system Ax b: ≤ 1. There is x such that Ax b. ≤ 2. There is y 0 such that AT y =0 but yT b< 0. ≥ LP-13 The intuition behind the precise form for 2. in the previous theorem lies in the proof that both cannot happen. The contradiction 0 = 0x =(yT A)x = yT (Ax)= yT b< 0 is obtained if AT y = 0 and yT b< 0.

9 Duality

Duality is the most important concept in linear programming. Duality allows to provide a proof of optimality. This is not only important algorithmically but also it leads to beautiful combinatorial statements. For example, consider the statement

In a graph, the smallest number of edges in a path between two spec- iﬁed vertices s and t is equal to the maximum number of s t cuts (i.e. − subsets of edges whose removal disconnects s and t).

This result is a direct consequence of duality for linear programming. Duality can be motivated by the problem of trying to ﬁnd lower bounds on the value of the optimal solution to a linear programming problem (if the problem is a maximization problem, then we would like to ﬁnd upper bounds). We consider problems in standard form:

min cT x s.t. Ax = b x 0 ≥ Suppose we wanted to obtain the best possible upper bound on the cost function. By multiplying each equation Amx = bm by some number ym and summing up the resulting equations, we obtain that yT Ax = bT y. if we impose that the coeﬃcient of T xj in the resulting inequality is less or equal to cj then b y must be a lower bound on the optimal value since xj is constrained to be nonnegative. To get the best possible lower bound, we want to solve the following problem:

max bT y s.t. AT y c ≤ This is another linear program. We call this one the dual of the original one, called the primal. As we just argued, solving this dual LP will give us a lower bound on the optimum value of the primal problem. Weak duality says precisely this: if we denote the optimum value of the primal by z, z = min cT x, and the optimum value of the dual by w, then w z. We will use Farkas’ lemma to prove strong duality which says ≤ that these quantities are in fact equal. We will also see that, in general, the dual of the dual is the problem.

LP-14 Example: z = min x1 + 2x2 + 4x3 x1 + x2 + 2x3 = 5 2x1 + x2 + 3x3 = 8

The ﬁrst equality gives a lower bound of 5 on the optimum value z, since x1 +2x2 + 4x3 x1 + x2 +2x3 = 5 because of nonnegativity of the xi. We can get an even better≥ lower bound by taking 3 times the ﬁrst equality minus the second one. This gives x1 +2x2 +3x3 = 7 x1 +2x2 +4x3, implying a lower bound of 7 on z. For 3 ≤ x =  2 , the objective function is precisely 7, implying optimality. The mechanism 0   of generating  lower bounds is formalized by the dual linear program:

max 5y1 + 8y2 y1 + 2y2 1 y + y ≤ 2 1 2 ≤ 2y + 3y 4 1 2 ≤

y1 represents the multiplier for the ﬁrst constraint and y2 the multiplier for the second constraint, This LP’s objective function also achieves a maximum value of 7 at y = 3 . 1 − We now formalize the notion of duality. Let P and D be the following pair of dual linear programs:

(P ) z = min cT x : Ax = b, x 0 { ≥ } (D) w = max bT y : AT y c . { ≤ } (P ) is called the primal linear program and (D) the dual linear program. In the proof below, we show that the dual of the dual is the primal. In other words, if one formulates (D) as a linear program in standard form (i.e. in the same form as (P )), its dual D(D) can be seen to be equivalent to the original primal (P ). In any statement, we may thus replace the roles of primal and dual without aﬀecting the statement. Proof: The dual problem D is equivalent to min bT y : AT y + Is = c,s 0 . Changing T + T T + T{− + ≥ } forms we get min b y +b y− : A y A y− +Is = c, and y ,y−,s 0 . Taking {− − ≥ } the dual of this we obtain: max cT x : A( x) b, A( x) b, I( x) 0 . But this is the same as min cT x : Ax{−= b, x 0− and≤ we − are− done.− ≤ − ≤ } 2 { ≥ } We have the following results relating w and z.

Lemma 9 (Weak Duality) z w. ≥ LP-15 Proof: Suppose x is primal feasible and y is dual feasible. Then, cT x yT Ax = yT b, ≥ thus z = min cT x : Ax = b, x 0 max bT y : AT y c = w. 2 From the{ preceding lemma≥ we} conclude ≥ { that the following≤ } cases are not possible (these are dual statements):

1. P is feasible and unbounded and D feasible.

2. P is feasible and D is feasible and unbounded. We should point out however that both the primal and the dual might be infeasible. To prove a stronger version of the weak duality lemma, let’s recall the following corollary of Farkas’ Lemma (Theorem 8): Corollary 10 Exactly one of the following is true:

1. x′ : A′x′ b′. ∃ ≤ T T 2. y′ 0:(A′) y′ =0 and (b′) y′ < 0. ∃ ≥ Theorem 11 (Strong Duality) If P or D is feasible then z = w.

Proof: We only need to show that z w. Assume without loss of generality (by duality) that P is feasible. If P is unbounded,≤ then by Weak Duality, we have that z = w = . Suppose P is bounded, and let x∗ be an optimal solution, i.e. Ax∗ = b, x∗ 0 −∞ T T T ≥ and c x∗ = z. We claim that y s.t. A y c and b y z. If so we are done. ∃ ≤ ≥ AT Suppose no such y exists. Then, by the preceding corollary, with A = , ′ bT − c x b′ = , x′ = y, y′ = , x 0, λ 0 such that z λ ∃ ≥ ≥ − Ax = λb and cT x < λz.

We have two cases Case 1: λ = 0. Since we can normalize by λ we can assume that λ = 1. This • means that x 0 such that Ax = b and cT x < z. But this is a contradiction ∃ ≥ with the optimality of x∗.

Case 2: λ = 0. This means that x 0 such that Ax = 0 and cT x< 0. If this • ∃ ≥ T is the case then µ 0, x∗ + µx is feasible for P and its cost is c (x∗ + µx)= T T ∀ ≥ c x∗ + µ(c x) < z, which is a contradiction. 2

LP-16 9.1 Rules for Taking Dual Problems If P is a minimization problem then D is a maximization problem. If P is a maximization problem then D is a minimization problem. In general, using the rules for transforming a linear program into standard form, we have that the dual of (P ): T T T z = min c1 x1 + c2 x2 + c3 x3 s.t.

A11x1 + A12x2 + A13x3 = b1 A x + A x + A x b 21 1 22 2 23 3 ≥ 2 A x + A x + A x b 31 1 32 2 33 3 ≤ 3 x 0 , x 0 , x UIS 1 ≥ 2 ≤ 3 (where UIS means “unrestricted in sign” to emphasize that no constraint is on the variable) is (D) T T T w = max b1 y1 + b2 y2 + b3 y3 s.t. T T T A11y1 + A21y2 + A31y3 c1 T T T ≤ A12y1 + A22y2 + A32y3 c2 T T T ≥ A13y1 + A23y2 + A33y3 = c3 y UIS , y 0 , y 0 1 2 ≥ 3 ≤ 10 Complementary Slackness

Let P and D be (P ) z = min cT x : Ax = b, x 0 { ≥ } (D) w = max bT y : AT y c , { ≤ } and let x be feasible in P , and y be fesible in D. Then, by weak duality, we know that cT x bT y. We call the diﬀerence cT x bT y the duality gap. Then we have that the ≥ − duality gap is zero iﬀ x is optimal in P , and y is optimal in D. That is, the duality gap can serve as a good measure of how close a feasible x and y are to the optimal solutions for P and D. The duality gap will be used in the description of the interior point method to monitor the progress towards optimality. It is convenient to write the dual of a linear program as

(D) w = max bT y : AT y + s = c for some s 0 { ≥ } Then we can write the duality gap as follows: cT x bT y = cT x xT AT y − − = xT (c AT y) − = xT s

LP-17 since AT y + s = c. The following theorem allows to check optimality of a primal and/or a dual solution.

Theorem 12 (Complementary Slackness) Let x∗, (y∗,s∗) be feasible for (P ), (D) respectively. The following are equivalent:

1. x∗ is an optimal solution to (P ) and (y∗,s∗) is an optimal solution to (D).

T 2. (s∗) x∗ =0.

3. x∗s∗ =0, j =1,...,n. j j ∀

4. If sj∗ > 0 then xj∗ =0.

Proof: T T T Suppose (1) holds, then, by strong duality, c x∗ = b y∗. Since c = A y∗ + s∗ and T T T T T Ax∗ = b, we get that (y∗) Ax∗ +(s∗) x∗ =(x∗) A y∗, and thus, (s∗) x∗ = 0 (i.e (2) holds). It follows, since x∗, s∗ 0, that x∗s∗ = 0, j = 1,... ,n (i.e. (3) holds). j j ≥ j j ∀ Hence, if s∗ > 0 then x∗ = 0, j =1,... ,n (i.e. (4) holds). The converse also holds, j j ∀ and thus the proof is complete. 2 In the example of section 9, the complementary slackness equations corresponding to the primal solution x = (3, 2, 0)T would be:

y1 +2y2 =1

y1 + y2 =2

Note that this implies that y = 3 and y = 1. Since this solution satisﬁes the 1 2 − other constraint of the dual, y is dual feasible, proving that x is an optimum solution to the primal (and therefore y is an optimum solution to the dual).

11 Size of a Linear Program

11.1 Size of the Input If we want to solve a Linear Program in polynomial time, we need to know what would that mean, i.e. what would the size of the input be. To this end we introduce two notions of the size of the input with respect to which the algorithm we present will run in polynomial time. The ﬁrst measure of the input size will be the size of a LP, but we will introduce a new measure L of a LP that will be easier to work with. Moreover, we have that L size(LP ), so that any algorithm running in time polynomial in L will also run in time≤ polynomial in size(LP).

LP-18 Let’s consider the linear program of the form:

min cT x s.t. Ax = b x 0 ≥ where we are given as inputs the coefficients of A (an m n matrix), b (an m 1 vector), and c (an n 1 vector), whith rationial entries. × × × We can further assume, without loss of generality, that the given coefficients are all integers, since any LP with rational coefficients can be easily transformed into an equivalent one with integer coefficients (just multiply everything by l.c.d.). In the rest of these notes, we assume that A, b, c have integer coefficients. For any integer n, we define its size as follows:

size(n) =1+△ log ( n + 1) ⌈ 2 | | ⌉ where the first 1 stands for the fact that we need one bit to store the sign of n, size(n) represents the number of bits needed to encode n in binary. Analogously, we define the size of a p 1 vector d, and of a p l matrix M as follows: × × p △ size(v) = i=1 size(vi) p △ l size(M) = i=1 j=1 size(mij) We are then ready to talk about the size of a LP. Definition 6 (Size of a linear program)

size(LP) =△ size(A) + size(b) + size(c).

A more convenient deﬁnition of the size of a linear program is given next. Deﬁnition 7

L =△ size(detmax) + size(bmax) + size(cmax)+ m + n where

detmax =△ max( det(A′) ) A′ | |

bmax =△ max( bi ) i | |

cmax =△ max( cj ) j | |

and A′ is any square submatrix of A.

LP-19 Proposition 13 L< size(LP), A, b, c. ∀ Before proving this result, we ﬁrst need the following lemma:

size(n) 1 Lemma 14 1. If n Z then n 2 − 1. ∈ | | ≤ − n size(v) n 2. If v Z then v v 2 − 1. ∈ ≤ 1 ≤ − n n size(A) n2 3. If A Z × then det(A) 2 − 1. ∈ | | ≤ − Proof:

1. By deﬁnition.

n n n size(vi) 1 size(v) n 2. 1+ v 1+ v 1 =1+ vi (1+ vi ) 2 − =2 − where ≤ i=1 | | ≤ i=1 | | ≤ i=1 we have used 1.

3. Let a ,...,a be the columns of A. Since det(A) represents the volume of the 1 n | | parallelepiped spanned by a1,...,an, we have

n det(A) a . | | ≤ i i=1 Hence, by 2,

n n n 2 size(ai) n size(A) n 1+ det(A) 1+ a (1 + a ) 2 − =2 − . | | ≤ i ≤ i ≤ i=1 i=1 i=1 2 We now prove Proposition 13. Proof: If B is a square submatrix of A then, by deﬁnition, size(B) size(A). Moreover, size(B) 1 ≤ by lemma 14, 1 + det(B) 2 − . Hence, | | ≤ log(1 + det(B) ) size(B) 1 < size(B) size(A). (1) ⌈ | | ⌉ ≤ − ≤ Let v Zp. Then size(v) size(max v )+ p 1= log(1 + max v ) + p. Hence, ∈ ≥ j | j| − ⌈ j | j| ⌉

size(b)+ size(c) log(1 + max cj ) + log(1 + max bi ) + m + n. (2) ≥ ⌈ j | | ⌉ ⌈ i | | ⌉ Combining equations (1) and (2), we obtain the desired result. 2

m+n L size(n) Remark 1 detmax bmax cmax 2 < 2 , since for any integer n, 2 > n . ∗ ∗ ∗ | | In what follows we will work with L as the size of the input to our algorithm.

LP-20 11.2 Size of the Output In order to even hope to solve a linear program in polynomial time, we better make sure that the solution is representable in size polynomial in L. We know already that if the LP is feasible, there is at least one vertex which is an optimal solution. Thus, when ﬁnding an optimal solution to the LP, it makes sense to restrict our attention to vertices only. The following theorem makes sure that vertices have a compact representation.

Theorem 15 Let x be a vertex of the polyhedron deﬁned by Ax = b, x 0. Then, ≥ p p p xT = 1 2 . . . n , q q q where p (i =1,...,n), q N, i ∈ and 0 p < 2L ≤ i 1 q < 2L. ≤ Proof: 1 Since x is a basic feasible solution, a basis B such that xB = AB− b and xN = 0. Thus, we can set p = 0, j N, and∃ focus our attention on the x ’s such that j ∀ ∈ j j B. We know by linear algebra that ∈

1 1 xB = AB− b = cof(AB)b det(AB)

where cof(AB) is the cofactor matrix of AB. Every entry of AB consists of a determinant of some submatrix of A. Let q = det(AB) , then q is an integer since AB has | | L integer components, q 1 since AB is invertible, and q detmax < 2 . Finally, note ≥ m ≤ L that pB = qxB = cof(AB)b , thus pi j=1 cof(AB)ij bj m detmax bmax < 2 . | | ≤ | || | ≤ 2

12 Complexity of linear programming

In this section, we show that linear programming is in NP co-NP. This will follow from duality and the estimates on the size of any vertex given∩ in the previous section. Let us deﬁne the following decision problem:

Deﬁnition 8 ( ) Input: LPIntegral A, b, c, and a rational number λ, Question: Is min cT x : Ax = b, x 0 λ? { ≥ } ≤

LP-21 Theorem 16 NP co-NP LP ∈ ∩ Proof: First, we prove that NP. LP ∈ If the linear program is feasible and bounded, the “certificate” for verification of T instances for which min c x : Ax = b, x 0 λ is a vertex x′ of Ax = b, x 0 T { ≥ } ≤ { ≥ } s.t. c x′ λ. This vertex x′ always exists since by assumption the minimum is finite. ≤ Given x′, it is easy to check in polynomial time whether Ax′ = b and x′ 0. We also need to show that the size of such a certificate is polynomially bounded≥ by the size of the input. This was shown in section 11.2. If the linear program is feasible and unbounded, then, by strong duality, the dual is infeasible. Using Farkas’ lemma on the dual, we obtain the existence ofx ˜: Ax˜ = 0, x˜ 0 and cT x˜ = 1 < 0. Our certificate in this case consists of both a vertex of Ax≥ = b, x 0 (to− show feasiblity) and a vertex of Ax = 0, x 0, cT x = 1 { ≥ } { ≥ − } (to show unboundedness if feasible). By choosing a vertex x′ of Ax = 0, x 0, T { ≥ c x = 1 , we insure that x′ has polynomial size (again, see Section 11.2). This− proves} that NP. (Notice that when the linear program is infeasible, LP ∈ the answer to is “no”, but we are not responsible to offer such an answer in order to show LPNP). LP ∈ Secondly, we show that co-NP, i.e. NP, where is defined as: Input: A, b, c, and aLP rational ∈ number λLP, ∈ LP Question: Is min cT x : Ax = b, x 0 >λ? { ≥ } If x : Ax = b, x 0 is nonempty, we can use strong duality to show that is { ≥ } LP indeed equivalent to: Input: A, b, c, and a rational number λ, Question: Is max bT y : AT y c >λ? which is also in NP, for{ the same reason≤ } as is. If the primal is infeasible, by Farkas’ lemmaLP we know the existence of a y s.t. AT y 0 and bT y = 1 < 0. This completes the proof of the theorem. 2 ≥ − 13 Solving a Liner Program in Polynomial Time

The ﬁrst polynomial-time algorithm for linear programming is the so-called ellipsoid algorithm which was proposed by Khachian in 1979 [6]. The ellipsoid algorithm was in fact ﬁrst developed for convex programming (of which linear programming is a special case) in a series of papers by the russian mathematicians A.Ju. Levin and, D.B. Judin and A.S. Nemirovskii, and is related to work of N.Z. Shor. Though of polynomial running time, the algorithm is impractical for linear programming. Nevertheless it has extensive theoretical applications in combinatorial optimization. For example, the stable set problem on the so-called perfect graphs can be solved in polynomial time using the ellipsoid algorithm. This is however a non-trivial non-combinatorial algorithm.

LP-22 In 1984, Karmarkar presented another polynomial-time algorithm for linear programming. His algorithm avoids the combinatorial complexity (inherent in the simplex algorithm) of the vertices, edges and faces of the polyhedron by staying well inside the polyhedron (see Figure 13). His algorithm lead to many other algorithms for linear programming based on similar ideas. These algorithms are known as interior point methods.

Figure 6: Exploring the interior of a convex body.

It still remains an open question whether there exists a strongly polynomial algorithm for linear programming, i.e. an algorithm whose running time depends on m and n and not on the size of any of the entries of A, b or c. In the rest of these notes, we discuss an interior-point method for linear programming and show its polynomiality. High-level description of an interior-point algorithm:

1. If x (current solution) is close to the boundary, then map the polyhedron onto another one s.t. x is well in the interior of the new polyhedron (see Figure 7).

2. Make a step in the transformed space.

3. Repeat (a) and(b) until we are close enough to an optimal solution.

Before we give description of the algorithm we give a theorem, the corollary of which will be a key tool used in determinig when we have reached an optimal solution.

LP-23 Theorem 17 Let x1, x2 be vertices of Ax = b, x 0. ≥ T T T T 2L If c x = c x then c x c x > 2− . 1 2 | 1 − 2| Proof: By Theorem 15, q , q , such that 1 q , q < 2L, and q x , q x Nn. Further- ∃ i 2 ≤ 1 2 1 1 2 2 ∈ more,

T T T T q1c x1 q2c x2 c x1 c x2 = | − | q − q 1 2 T T q q (c x c x ) = 1 2 1 − 2 q q 1 2 1 T T since c x1 c x2 = 0, q1, q2 1 ≥ q1q2 − ≥

1 2L L > =2− since q , q < 2 . 2L2L 1 2 2

Corollary 18 Assume z = min cT x : Ax = b, x 0 . { ≥ } polyhedron P T 2L Assume x is feasible to P , and such that c x z +2− . ≤ T T Then, any vertex x′ such that c x′ c x is an optimal solution of the LP. ≤ Proof: T Suppose x′ is not optimal. Then, x∗, an optimal vertex, such that c x∗ = z. T T ∃ Since x′ is not optimal, c x′ = c x∗, and by Theorem 17 T T 2L c x′ c x∗ > 2− ⇒ − T T 2L c x′ > c x∗ +2− ⇒ 2L = Z +2− cT x by definition of x ≥ T c x′ by definition of x′ T ≥ T c x′ > c x′, ⇒ a contradiction. 2 What this corollary tells us is that we do not need to be very precise when choosing an optimal vertex. More precisely we only need to compute the objective function 2L with error less than 2− . If we find a vertex that is within that margin of error, then it will be optimal.

LP-24 x’ x

P P’

Figure 7: A centering mapping. If x is close to the boundary, we map the polyhedron P onto another one P ′, s.t. the image x′ of x is closer to the center of P ′.

13.1 Ye’s Interior Point Algorithm In the rest of these notes we present Ye’s [9] interior point algorithm for linear programming. Ye’s algorithm (among several others) achieves the best known asymptotic running time in the literature, and our presentation incorporates some simpliﬁcations made by Freund [3]. We are going to consider the following linear programming problem:

minimize Z = cT x (P )  subject to Ax = b,  x 0 ≥ and its dual  maximize W = bT y T (D)  subject to A y + s = c,  s 0. ≥ The algorithm is primal-dual, meaning that it simultaneously solves both the primal and dual problems. It keeps track of a primal solution x and a vector of dual slacks s (i.e. y : AT y = c s) such that x > 0 and s > 0. The basic idea of this ∃ − algorithm is to stay away from the boundaries of the polyhedron (the hyperplanes x 0 and s 0, j =1, 2,...,n) while approaching optimality. In other words, we j ≥ j ≥ want to make the duality gap

cT x bT y = xT s> 0 − LP-25 very small but stay away from the boundaries. Two tools will be used to achieve this goal in polynomial time. Tool 1: Scaling (see Figure 7) Scaling is a crucial ingredient in interior point methods. The two types of scaling commonly used are projective scaling (the one used by Karmarkar) and aﬃne scaling (the one we are going to use). T Suppose the current iterate is x> 0 and s> 0, where x =(x1, x2,..., xn) , then the aﬃne scaling maps x to x′ as follows.

x1 x1 x1 x2  x2   x2  .  .  x =   x′ =    .  −→  .       .       .     xn   xn       xn .     Notice this transformation maps x to e = (1,..., 1)T . 1 We can express the scaling transformation in matrix form as x′ = X− x or x = Xx′, where x1 0 0 . . . 0  0 x2 0 . . . 0  . . . X =  . . .       0 0 . . . xn 1 0   −   0 0 . . . 0 xn   . Using matrix notation we can rewrite the linear program (P) in terms of the transformed variables as:

T minimize Z = c Xx′

subject to AXx′ = b,

x′ 0. ≥ If we deﬁne c = Xc (note that X = XT ) and A = AX we can get a linear program in the original form as follows.

T minimize Z = c x′

subject to Ax′ = b,

x′ 0. ≥ We can also write the dual problem (D) as: maximize W = bT y subject to (AX)T y + Xs = c, Xs 0 ≥ LP-26 or, equivalently,

maximize W = bT y T subject to A y + s′ = c,

s′ 0 ≥ where s′ = Xs, i.e. s1x1  s2x2  s′ = .      .     s x   n n .   One can easily see that

x s = x′ s′ j 1,...,n (3) j j j j ∀ ∈{ } T and, therefore, the duality gap x s = j xjsj remains unchanged under aﬃne scaling. As a consequence, we will see later that one can always work equivalently in the transformed space. Tool 2: Potential Function Our potential function is designed to measure how small the duality gap is and how far the current iterate is away from the boundaries. In fact we are going to use the following “logarithmic barrier function”.

Deﬁnition 9 (Potential Function, G(x, s))

n G(x, s) =△ q ln(xT s) ln(x s ), for some q, − j j j=1 where q is a parameter that must be chosen appropriately. Note that the ﬁrst term goes to as the duality gap tends to 0, and the second −∞ term goes to + as xi 0 or si 0 for some i. Two questions arise immediately concerning this∞ potential→ function.→ Question 1: How do we choose q?

n 1 Lemma 19 Let x,s > 0 be vectors in R × . Then

n n ln xT s ln x s n ln n. − j j ≥ j=1

LP-27 Proof: Given any n positive numbers t1,...,tn, we know that their geometric mean does not exceed their arithmetic mean, i.e.

1/n n 1 n t t  j ≤ n  j j=1 j=1 .     Taking the logarithms of both sides we have

1 n n ln t ln t ln n. n  j ≤  j − j=1 j=1     Rearranging this inequality we get

n n n ln t ln t n ln n.  j −  j ≥ j=1 j=1     (In fact the last inequality can be derived directly from the concavity of the logarithmic function). The lemma follows if we set tj = xjsj. 2 Since our objective is that G as xT s 0 (since our primary goal is to get close to optimality), according to→ Lemma −∞ 19, we→ should choose some q > n (notice that ln xT s as xT s 0) . In particular, if we choose q = n + 1, the algorithm will terminate→ −∞ after O(nL)→ iterations. In fact we are going to set q = n + √n, which gives us the smallest number — O(√nL) — of iterations by this method. Question 2: When can we stop? T 2L T T T T 2L Suppose that x s 2− , then c x Z c x b y = x s 2− , where Z is ≤ − ≤ − ≤ the optimum value to the primal problem. From Corollary 18, the following claim follows immediately.

T 2L T T Claim 20 If x s 2− , then any vertex x∗ satisfying c x∗ c x is optimal. ≤ ≤

In order to ﬁnd x∗ from x, two methods can be used. One is based on purely algebraic techniques (but is a bit cumbersome to describe), while the other (the cleanest one in literature) is based upon basis reduction for lattices. We shall not elaborate on this topic, although we’ll get back to this issue when discussing basis reduction in lattices.

Lemma 21 Let x, s be feasible primal-dual vectors such that G(x, s) k√nL for ≤ − some constant k. Then T kL x s < e− .

LP-28 Proof: By the deﬁnition of G(x, s) and the previous theorem we have:

k√nL G(x, s) − ≥ n = (n + √n) ln xT s ln x s − j j j=1 √n ln xT s + n ln n. ≥ Rearranging we obtain

ln xT s kL √n ln n ≤ − − < kL. − Therefore T kL x s < e− . 2 The previous lemma and claim tell us that we can stop whenever G(x, s) 2√nL. In practice, the algorithm can terminate even earlier, so it is a good idea to≤ − check from time to time if we can get the optimal solution right away. Please notice that according to Equation (3) the aﬃne transformation does not change the value of the potential function. Hence we can work either in the original space or in the transformed space when we talk about the potential function.

14 Description of Ye’s Interior Point Algorithm

Initialization: Set i = 0. Choose x0 > 0, s0 > 0, and y0 such that Ax0 = b, AT y0 + s0 = c and G(x0,s0)= O(√nL). (Details are not covered in class but can be found in the appendix. The general idea is as follows. By augmenting the linear program with additional variables, it is easy to obtain a feasible solution. Moreover, by carefully choosing the augmented linear program, it is possible to have feasible primal and dual solutions x and s such L that all xj’s and sj’s are large (say 2 ). This can be seen to result in a potential of O(√nL).) Iteration: while G(xi,si) > 2√nL − i either a primal step (changing x only) i+1 i+1 i to get (x ,s ) do  or a dual step (changing s only)  i := i +1

 i i The iterative step is as follows. Aﬃne scaling maps (x ,s ) to (e, s′). In this transformed space, the point is far away from the boundaries. Either a dual or

LP-29 a1 a 2 g

g−d

Null space of A d {x: Ax = 0}

Figure 8: Null space of A and gradient direction g. primal step occurs, giving (˜x, s˜) and reducing the potential function. The point is then mapped back to the original space, resulting in (xi+1,si+1). Next, we are going to describe precisely how the primal or dual step is made such that 7 G(xi+1,si+1) G(xi,si) < 0 − ≤ −120 holds for either a primal or dual step, yielding an O(√nL) total number of iterations. In order to ﬁnd the new point (˜x, s˜) given the current iterate (e, s′) (remember we are working in the transformed space), we compute the gradient of the potential function. This is the direction along which the value of the potential function changes at the highest rate. Let g denote the gradient. Recall that (e, s′) is the map of the current iterate, we obtain

g = G(x, s) ′ ∇x |(e,s ) 1/x1 q . = s  .  T . x s −  1/x  ′  n (e,s )   q ′ = T s e (4) e s′ − We would like to maximize the change in G, so we would like to move in the direction of g. However, we must insure the new point is still feasible (i.e. Ax˜ = b). − Let d be the projection of g onto the null space x : Ax = 0 of A. Thus, we will move in the direction of d. { } −

T 1 Claim 22 d =(I A(A A )− A)g. − LP-30 Proof: Since g d is orthogonal to the null space of A, it must be the combination of − some row vectors of A. Hence we have Ad =0 T w, s.t. A w = g d. ∃ − This implies

T A w = g d T − (normal equations).  (A A )w = Ag  Solving the normal equations, we get

T 1 w =(A A )− Ag and T T 1 T T 1 d = g A (A A )− Ag =(I A (A A )− A)g. − − 2 A potential problem arises if g is nearly perpendicular to the null space of A. In this case, d will be very small, and each primal step will not reduce the potential greatly. Instead,|| || we will perform a dual step. In particular, if d = d = √dT d 0.4, we make a primal step as follows. || || || ||2 ≥ 1 x˜ = e d − 4 d || || s˜ = s′.

Claim 23 x>˜ 0.

Proof: x˜ =1 1 dj 3 > 0. 2 j − 4 d ≥ 4 This claim|| insures|| that the new iterate is still an interior point. For the similar reason, we will see thats> ˜ 0 when we make a dual step.

7 Proposition 24 When a primal step is made, G(˜x, s˜) G(e, s′) . − ≤ − 120 If d < 0.4, we make a dual step. Again, we calculate the gradient || || h = G(x, s) ′ ∇s |(e,s ) 1/s′ q 1 . = T e  .  (5) e s′ −  1/s′   n    LP-31 Notice that hj = gj/sj, thus h and g can be seen to be approximately in the same direction. Suppose the current dual feasible solution is y′,s′ such that

T A y′ + s′ = c.

Again, we restrict the solution to be feasible, so

AT y +˜s = c T s˜ s′ = A (y′ y) − − Thus, in the dual space, we move perpendicular to the null space and in the direction of (g d). −Thus,− we have s˜ = s′ (g d)µ − − T For any µ, y A y +˜s = c ∃ eT s′ T So, we can choose µ = q and get A (y′ + µw)+˜s = c. Therefore,

T e s′ s˜ = s′ (g d) − q − T e s′ s′ = s′ (q T e d) − q e s′ − − eT s = ′ (d + e) q x˜ = x′ = e.

One can show thats> ˜ 0 as we did in Claim 23. So such move is legal.

1 Proposition 25 When a dual step is made, G(˜x, s˜) G(e, s′) − ≤ − 6 According to these two propositions, the potential function decreases by a constant amount at each step. So if we start from an initial interior point (x0,s0) with G(x0,s0) = O(√nL), then after O(√nL) iterations we will obtain another interior point (xj,sj) with G(xj,sj) k√nL. From Lemma 21, we know that the duality ≤ − gap (xj)T sj satisﬁes j T j kL (x ) s 2− , ≤ and the algorithm terminates by that time. Moreover, each iteration requires O(n3) operations. Indeed, in each iteration, the only non-trivial task is the computation of the projected gradient d. This can be done by solving the linear system (A¯A¯T )w = Ag¯ in O(n3) time using Gaussian elimination. Therefore, the overall time complexity of this algorithm is O(n3.5L). By using approximate solutions to the linear systems, we can obtain O(n2.5) time per iteration, and total time O(n3L).

LP-32 15 Analysis of the Potential Function

In this section, we prove the two propositions of the previous section, which concludes the analysis of Ye’s algorithm. Proof of Proposition 24: 1 G(˜x, s˜) G(e, s′) = G(e d, s˜) G(e, s′) − − 4 d − || || T n n T d s′ dj = q ln e s′ ln 1 ln sj′ − 4 d − − 4 d − − || || j=1 || || j=1 n n T q ln e s′ + ln1+ ln s′ − j j=1 j=1 T n d s′ dj = q ln 1 T ln 1 . − 4 d e s′ − − 4 d || || j=1 || || Using the relation

x2 x ln(1 x) x (6) − − 2(1 a) ≤ − ≤ − − which holds for x a< 1, we get: | | ≤ T n n 2 qd s′ dj dj ′ G(˜x, s˜) G(e, s ) T + + 2 for a =1/4 − ≤ −4 d e s′ 4 d 16 d 2(3/4) || || j=1 || || j=1 || || qdT s eT d 1 = ′ + + −4 d eT s 4 d 24 || || ′ || || 1 q T 1 = (e s′) d + 4 d − eT s 24 || || ′ 1 1 = ( g)T d + 4 d − 24 || || d 2 1 = || || + −4 d 24 || || d 1 = || || + − 4 24 1 1 + ≤ −10 24 7 = . −120 Note that gT d = d 2, since d is the projection of g. (This is where we use the || || fact that d is the projected gradient!) 2 Before proving Proposition 25, we need the following lemma.

LP-33 Lemma 26 n eT s˜ 2 ln(˜s ) n ln( ) − . j − n ≥ 15 j=1 Proof: ∆ Using the equalitys ˜ = q (e + d) and Equation 6, which holds for x a< 1, we see that | | ≤ T T n ln(˜s ) n ln( e s˜) = n ln( ∆ (1 + d )) n ln( ∆ (1 + e d )) j=1 j − n j=1 q j − q n 2 T n (d dj ) n e d ≥ j=1 j − 2(3/5) − n d 2 || || ≥ − 6/5

2 −15 ≥ 2 Proof of Proposition 25: Using Lemma 26 and the inequality n eT s ln(s ) n ln( ), j ≤ n j=1 which follows from the concavity of the logarithm function, we have

eT s˜ n n G(e, s˜) G(e, s′) = q ln( T ′ ) ln(˜s )+ ln(s′ ) − e s − j=1 j j=1 j eT s˜ 2 eT s˜ eT s′ q ln( ′ )+ n ln( )+ n ln( ) ≤ eT s 15 − n n

2 eT s˜ = 15 + √n ln( eT s′ ) On the other hand, ∆ eT s˜ = (n + eT d) q T and recall that ∆ = e s′, eT s˜ 1 1 T √ T = (n + e d) (n +0.4 n), e s′ q ≤ n + √n since, by Cauchy-Schwartz inequality, eT d e d = √n d . Combining the above inequalities yields | | ≤ || |||| || || || 2 0.6√n G(e, s˜) G(e, s′) + √n ln(1 ) − ≤ 15 − n+√n

2 0.6n ≤ 15 − n+√n 2 3 = 1 ≤ 15 − 10 − 6 LP-34 since n + √n 2n. 2 ≤ This completes the analysis of Ye’s algorithm.

16 Bit Complexity

Throughout the presentation of the algorithm, we assumed that all operations can be performed exactly. This is a fairly unrealistic assumption. For example, notice that d might be irrational since it involves a square root. However, none of the thresholds we set were crucial. We could for example test whether d 0.4 or d 0.399. To test this, we need to compute only a few bits of d. ≥ Also, if ≤ we perform a primal step (i.e. d 0.4) and compute the ﬁrst few bits of d so ≥ that the resulting approximation d ap satisﬁes (4/5) d d ap d then if we go through the analysis of the primal step performed in Proposi ≤tion 1,≤ we obtain that the reduction in the potential function is at least 19/352 instead of the previous 7/120. Hence, by rounding d we can still maintain a constant decrease in the potential function. Another potential problem is when using Gaussian elimination to compute the projected gradient. We mentioned that Gaussian elimination requires O(n3) arithmetic operations but we need to show that, during the computation, the numbers involved have polynomial size. For that purpose, consider the use of Gaussian elimination to solve a system Ax = b where

(1) (1) (1) a11 a12 ... a1n (1) (1) (1) (1)  a21 a22 ... a2n  A = A = . . . .  . . .. .   . . . .   (1) (1)   a a ... a(1)   m1 m2 mn    Assume that a11 = 0 (otherwise, we can permute rows or columns). In the ﬁrst (1) (1) iteration, we substract ai1 /a11 times the ﬁrst row from row i where i = 2,...,m, resulting in the following matrix:

(2) (2) (2) a11 a12 ... a1n (2) (2) (2)  0 a22 ... a2n  A = . . . .  . . .. .   . . . .   (2)   0 a ... a(2)   m2 mn    (i+1) (i) (i) (i) In general, A is obtained by subtracting aji /aii times row i from row j of A for j = i +1,...,m.

(i) Theorem 27 For all i j, k, ajk can be written in the form det(B)/ det(C) where B and C are some submatrices≤ of A.

LP-35 Proof: (i) Let Bi denote the i i submatrix of A consisting of the first i entries of the first i rows. Let B(i) denote× the i i submatrix of A(i) consisting of the first i 1 rows jk × − and row j, and the first i 1 columns and column k. Since B and B(i) are upper − i jk triangular matrices, their determinants are the products of the entries along the main diagonal and, as a result, we have:

(i) det(Bi) aii = det(Bi 1) − and (i) (i) det(Bjk ) ajk = . det(Bi 1) − Moreover, remember that row operations do not aﬀect the determinants and, hence, (i) the determinants of Bjk and Bi 1 are also determinants of submatrices of the original − matrix A. 2 Using the fact that the size of the determinant of any submatrix of A is at most the size of the matrix A, we obtain that all numbers occuring during Gaussian elimination require only O(L) bits. Finally, we need to round the current iterates x, y and s to O(L) bits. Otherwise, these vectors would require a constantly increasing number of bits as we iterate. By rounding up x and s, we insure that these vectors are still strictly positive. It is fairly easy to check that this rounding does not change the potential function by a signiﬁcant amount and so the analysis of the algorithm is still valid. Notice that now the primal and dual constraints might be slightly violated but this can be taken care of in the rounding step.

A Transformation for the Interior Point Algorithm

In this appendix, we show how a pair of dual linear programs

Min cT x Max bT y (P ) s.t. Ax = b (D) s.t. AT y + s = c x 0 s 0 ≥ ≥

can be transformed so that we know a strictly feasible primal solution x0 and a strictly feasible vector of dual slacks s0 such that G(x0; s0)= O(√nL) where

n G(x; s)= q ln(xT s) ln(x s ) − j j j=1 and q = n + √n.

LP-36 Consider the pair of dual linear programs:

T Min c x + kcxn+1 2L (P ′) s.t. Ax + (b 2 Ae)xn+1 = b (24Le c)T x − + 24Lx = k − n+2 b x 0 x 0 x 0 ≥ n+1 ≥ n+2 ≥ and

T Min b y + kbym+1 T 4L (D′) s.t. A y + (2 e c)ym+1 + s = c 2L T − (b 2 Ae) y + sn+1 = kc − 4L 2 ym+1 + sn+2 = 0 s, s , s 0 n+1 n+2 ≥ 6L 2L T where kb = 2 (n + 1) 2 c e is chosen in such a way that x′ = (x, xn+1, xn+2) = 2L 2L − 6L (2 e, 1, 2 ) is a (strict) feasible solution to (P ′) and kc =2 . Notice that (y′,s′)= 4L 4L (y,y ,s,s ,s )=(0, 1, 2 e, k , 2 ) is a feasible solution to (D′) with s′ > 0. m+1 n+1 n+2 − c x′ and (y′,s′) serve as our initial feasible solutions. We have to show:

1. G(x′; s′)= O(√n′L) where n′ = n + 2,

2. the pair (P ′) (D′) is equivalent to (P ) (D), − −

3. the input size L′ for (P ′) as deﬁned in the lecture notes does not increase too much.

The proofs of these statements are simple but heavily use the definition of L and the fact that vertices have all components bounded by 2L. 6L We first show 1. Notice first that xj′ sj′ =2 for all j, implying that

n′ T G(x′; s′) = (n′ + √n ) ln(x′ s′) ln(x′ s′ ) ′ − j j j=1 6L 6L = (n′ + √n′) ln(2 n′) n′ ln(2 ) 6L − = √n′ ln(2 )+(n′ + √n′) ln(n′)

= O(√n′L)

In order to show that (P ′) (D′) are equivalent to (P ) (D), we consider an − − optimal solution x∗ to (P ) and an optimal solution (y∗,s∗) to (D) (the case where (P ) or (D) is infeasible is considered in the problem set). Without loss of generality, we can assume that x∗ and (y∗,s∗) are vertices of the corresponding polyhedra. In L particular, this means that x∗, y∗ ,s∗ < 2 . j | j | j LP-37 4L T 4L Proposition 28 Let x′ =(x∗, 0, (kb (2 e c) x∗)/2 ) and let (y′,s′)=(y∗, 0,,s∗,kc 2L T − − − (b 2 Ae) y∗, 0). Then − 1. x′ is a feasible solution to (P ′) with xn′ +2 > 0,

2. (y′,s′) is a feasible solution to (D′) with sn′ +1 > 0,

3. x′ and (y′,s′) satisfy complementary slackness, i.e. they constitute a pair of optimal solutions for (P ′) (D′). − Proof:

To show that x′ is a feasible solution to (P ′) with xn′ +2 > 0, we only need to 4L T show that k (2 e c) x∗ > 0 (the reader can easily verify that x′ satisfy all the b − − equalities deﬁning the feasible region of (P ′)). This follows from the fact that 4L T 4L L L 5L 2L 6L (2 e c) x∗ n(2 +2 )2 = n(2 +2 ) < n2 − ≤ and 6L 2L T 6L 2L 6L 6L 3L 6L kb =2 (n + 1) 2 c e 2 (n + 1) 2 n max cj 2 n +2 2 > n2 − ≥ − j | | ≥ − where we have used the deﬁnition of L and the fact that vertices have all their entries bounded by 2L.

To show that (y′,s′) is a feasible solution to (D′) with sn′ +1 > 0, we only need to 2L T show that k (b 2 Ae) y∗ > 0. This is true since c − − 2L T T 2L T T (b 2 Ae) y∗ b y∗ 2 e A y∗ − ≤ − L 2L L m max bi 2 +2 nm max aij 2 ≤ i | | i,j | | 2L 4L 6L = 2 +2 < 2 = kc.

x′ and (y′,s′) satisfy complementary slackness since T x∗ s∗ = 0 by optimality of x∗ and (y∗,s∗) for (P ) and (D) • x′ s′ = 0 and • n+1 n+1 x′ s′ = 0. • n+2 n+2 2 This proposition shows that, from an optimal solution to (P ) (D), we can easily − construct an optimal solution to (P ′) (D′) of the same cost. Since this solution − has sn′ +1 > 0, any optimal solutionx ˆ to (P ′) must havex ˆn+1 = 0. Moreover, since xn′ +2 > 0, any optimal solution (ˆy, sˆ) to (D′) must satisfys ˆn+2 = 0 and, as a result, yˆ = 0. Hence, from any optimal solution to (P ′) (D′), we can easily deduce an m+1 − optimal solution to (P ) (D). This shows the equivalence between (P ) (D) and − − (P ′) (D′). − By some tedious but straightforward calculations, it is possible to show that L′ (corresponding to (P ′) (D′)) is at most 24L. In other words, (P ) (D)and(P ′) (D′) have equivalent sizes. − − −

LP-38 References

[1] V. Chvatal. Linear Programming. W.H. Freeman and Company, 1983.

[2] G. Dantzig. Maximization of a linear function of variables subject to linear inequalities. In T. Koopmans, editor, Activity Analysis of Production and Allocation, pages 339–347. John Wiley & Sons, Inc., 1951.

[3] R. M. Freund. Polynomial-time algorithms for linear programming based only on primal scaling and project gradients of a potential function. Mathematical Programming, 51:203–222, 1991.

[4] D. Goldfarb and M. Todd. Linear programming. In Handbook in Operations Re- search and Management Science, volume 1, pages 73–170. Elsevier Science Pub- lishers B.V., 1989.

[5] C. C. Gonzaga. Path-following methods for linear programming. SIAM Review, 34:167–224, 1992.

[6] L. Khachian. A polynomial algorithm for linear programming. Doklady Akad. Nauk USSR, 244(5):1093–1096, 1979.

[7] K. Murty. Linear Programming. John Wiley & Sons, 1983.

[8] A. Schrijver. Theory of Linear and Integer Programming. John Wiley & Sons, 1986.

[9] Y. Ye. An O(n3L) potential reduction algorithm for linear programming. Math- ematical Programming, 50:239–258, 1991.

LP-39 LP-40