Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 1

Conic Duality

Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A.

http://www.stanford.edu/˜yyye Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 2

Vectors and Norms

• Real numbers: R, R+, int R+ n n n • n-dimensional Euclidean space R , R+, int R+

• Component-wise: x ≥ y means xj ≥ yj for j =1, 2, ..., n

• 0: vector of all zeros; and e: vector of all ones

• Inner-product of two vectors: n T x • y := x y = xj yj j=1

√ T • Euclidean norm: x2 = x x, Infinity-norm: x∞ =max{|x1|, |x2|, ..., |xn|}, 1/p n p p-norm: xp = j=1 |xj | Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 3

∗ 1 1 • The dual of the p norm, denoted by . , is the q norm, where p + q =1

• Column vector:

x =(x1; x2; ...; xn)

Row vector:

x =(x1,x2,...,xn)

T • Transpose operation: A

• A set of vectors a1, ..., am is said to be linearly dependent if there are scalars λ1, ..., λm, not all zero, such that the linear combination m λiai = 0 i=1

n • A linearly independent set of vectors that span R is a basis. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 4

Hyper plane and Half-spaces

n H = {x : ax = ajxj = b} j=1 n + H = {x : ax = aj xj ≤ b} j=1 n − H = {x : ax = ajxj ≥ b} j=1 Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 5

y

(0,3) 3x+5y>15

3x+5y=15

3x+5y<15

0 (5,0) x

Figure 1: Plane and Half-Spaces Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 6

Matrices and Norms

m×n • Matrix: R , ith row: ai., jth column: a.j , ijth element: aij

• AI denotes the submatrix of A whose rows belong to index set I, AJ denotes the submatrix whose columns belong to index set J, AIJ denotes the submatrix whose rows belong to index set I and columns belong to index set J.

• Determinant: det(A), Trace: tr(A)

• The operator norm of A, Ax2 A2 := max 0=x∈Rn x2

• All-zero matrix: 0, and identity matrix: I T • Symmetric matrix: Q = Q Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 7

T • Positive Definite: Q  0 iff x Qx > 0, for all x = 0

T • Positive Semidefinite: Q  0 iff x Qx ≥ 0, for all x

• Null space and Row space of matrix A:

Theorem 1 The null space and row space of a matrix are perpenticular to each other, that is,

T T x s =0, ∀ Ax = 0 and s = A y. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 8

Symmetric Matrix Space

n • n-dimensional symmetric matrix space: S • Inner Product: T X • Y = trX Y = Xi,jYi,j i,j

• Frobenius norm: √ T Xf = trX X

n n • Positive semidefinite matrix set: S+, Positive definite matrix set: int S+ Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 9

• Decomposition of Symmetric Positive Semidefinite Matrices: r T X = xixi i=1

T where r is the rank of X, and xi xj =0for i = j.

• Let X be a positive semidefinite matrix of rank r, A be any given symmetric matrix. Then, there is a decomposition of X r T X = xjxj , j=1

such that for all j,

T T xj Axj = A • (xjxj )=A • X/r. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 10

Affine and Convex Combination

n S ⊂ R is affine if

[ x, y ∈ S and α ∈ R ]=⇒ αx +(1− α)y ∈ S.

n When x and y are two distinct points in R and α runs over R , {z : z = αx +(1− α)y} is the line set determined by x and y.

When 0 ≤ α ≤ 1, it is called the convex combination of x and y and it is the line segment between x and y. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 11

Convex Sets

• Set notations: x ∈ Ω, y ∈ Ω S ∪ T , S ∩ T

1 2 • Ω is said to be a convex set if for every x , x ∈ Ω and every real number 1 2 α ∈ [0, 1], the point αx +(1− α)x ∈ Ω.

• Intersection of convex sets is convex; the convex hull of a set Ω is the intersection of all convex sets containing Ω

• A point in a set is called an extreme point of the set if it cannot be represented as the convex combination of two distinct points of the set.

• A set is a polyhedral set if it has finitely many extreme points. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 12

Set Combinations

Let C1 and C2 be convex sets in a same space. Then,

• C1 ∩ C2 is convex.

• C1 + C2 is convex, where

C1 + C2 = {b1 + b2 : b1 ∈ C1 and b2 ∈ C2}.

• C1 ⊕ C2 is convex, where

C1 ⊕ C2 = {(b1; b2): b1 ∈ C1 and b2 ∈ C2}. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 13

Cones

• A set K is a cone if x ∈ K implies αx ∈ K for all α>0

• A convex cone is cone and it’s also a convex-set.

• Dual cone: ∗ K := {y : y • x ≥ 0 for all x ∈ K} ∗ −K is also called the polar of K.

• The dual of a cone is always a closed convex cone. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 14

Cone Examples

• Example 2.1: The n-dimensional non-negative orthant, n n R+ = {x ∈R : x ≥ 0}, is a convex cone; and it’s self dual. n • Example 2.2: The set of all positive semi-definite symmetric matrices in S , n S+, is a convex cone, called the positive semi-definite matrix cone; and it’s self dual.

n n • Example 2.3: The set {x ∈R : x1 ≥x−1}, N2 , is a convex cone in n R called the second-order (norm) cone; and it’s self dual.

n n • Example 2.4: The set {x ∈R : x1 ≥x−1p}, Np , is a convex cone in n R for p ≥ 1 called the p-order (norm) cone; and its dual is the q-order cone 1 1 where p + q =1. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 15

Cone and Dual Facts

Let K1 and K2 be both closed convex cones. Then

∗ ∗ i) (K1 ) = K1. ∗ ∗ ii) K1 ⊂ K2 =⇒ K2 ⊂ K1 . ∗ ∗ ∗ iii) (K1 ⊕ K2) = K1 ⊕ K2 . ∗ ∗ ∗ iv) (K1 + K2) = K1 ∩ K2 . ∗ ∗ ∗ v) (K1 ∩ K2) = K1 + K2 . Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 16

Convex Polyhedral Cones I

• A cone K is (convex) polyhedral if its intersection with a hyperplane is a polyhedral set.

• A convex cone K is polyhedral if and only if K can be represented by

K = {x : Ax ≤ 0} or {x : x = Ay, y ≥ 0}

for some matrix A. In the latter case, K is generated by the columns of A.

• The nonnegative orthant is a polyhedral cone but the second-order cone is not polyhedral. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 17

Polyhedral Cone Nonpolyhedral Cone

Figure 2: Polyhedral and non-polyhedral cones. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 18

Convex Polyhedral Cones II

It has been proved that for cones the concepts of “polyhedral” and “finitely generated” are equivalent according to the following theorem.

Theorem 2 (Minkowski and Weyl) A convex cone C is polyhedral if and only if it is finitely generated, that is, the cone is generated by a finite number of vectors b1,...,bm: m C = cone(b1, ..., bm):= biyi : yi ≥ 0,i=1, ..., m . i=1 Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 19

Caratheodory’s« theorem

The following theorem states that a polyhedral cone can be generated by a set of basic directional vectors.

Theorem 3 Let convex polyhedral cone C = cone(b1, ..., bm) and x ∈ C. x ∈ (b , ..., b ) b b Then, cone i1 id for some linearly independent vectors i1 ,..., id chosen from b1,...,bm.

Some times we even have: ⎧ ⎛ ⎞ ⎫ ⎧⎛ ⎞ ⎛ ⎞ ⎫ ⎨ −21 ⎬ ⎨ 1 2 ⎬ 2 ⎝ ⎠ ⎝ ⎠ ⎝ ⎠ x ∈R+ : x ≤ 0 = y1 + y2 : y1,y2 ≥ 0 . ⎩ 1 −2 ⎭ ⎩ 2 1 ⎭ Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 20

(1,2) 1

2 (2,1)

1 2

Figure 3: Representations of a polyhedral cone. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 21

Separating hyperplane theorem

The most important theorem about the convex set is the following separating hyperplane theorem (Figure 4).

n Theorem 4 (Separating hyperplane theorem) Let C ⊂E, where E is either R n or S , be a closed convex set and let b be a point exterior to C. Then there is a vector a ∈Esuch that a • b > sup a • x x∈C where a is the norm direction of the hyperplane. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 22

C

b -a

Figure 4: Illustration of the separating hyperplane theorem; an exterior point b is separated by a hyperplane from a convex set C. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 23

Examples

Let C be a unit circle centered at point (1; 1). That is, 2 2 2 C = {x ∈R :(x1 − 1) +(x2 − 1) ≤ 1}.Ifb = (2; 0), a =(−1; 1) is a separating hyperplane vector.

If b =(0;−1), a = (0; 1) is a separating hyperplane vector. It is worth noting that these separating hyperplanes are not unique. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 24

Farkas’ Lemma for Polyhedral Cone

m×n m Theorem 5 Let A ∈R and b ∈R . Then, the system n {x : Ax = b, x ∈R+} has a feasible solution x if and only if that T n T T {y : −A y ∈R+, b y > 0, (b y =1)} has no feasible solution. T T A vector y, with A y ≤ 0 and b y > 0, is called a infeasibility certificate for the system {x : Ax = b, x ≥ 0}.

Example: Let A =(1, 1) and b = −1. Then, y = −1 is an infeasibility certificate for {x : Ax = b, x ≥ 0}. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 25

Alternative Systems

Farkas’ lemma is also called the alternative theorem, that is, exactly one of the two systems: {x : Ax = b, x ≥ 0} and {y : AT y ≤ 0, bT y > 0, (bT y =1)}, is feasible. m Geometrically, Farkas’ lemma means that if a vector b ∈R does not belong to the cone generated by a.1, ..., a.n, then there is a hyperplane separating b from cone(a.1, ..., a.n), that is, b ∈ C := {Ax : x ≥ 0}, which is a closed convex set(?). Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 26

Proof

Let {x : Ax = b, x ≥ 0} have a feasible solution, say x¯. Then, T T {y : A y ≤ 0, b y > 0} is infeasible, since otherwise, 0 < bT y =(Ax)T y = xT (AT y) ≤ 0

T since x ≥ 0 and A y ≤ 0.

Now let {x : Ax = b, x ≥ 0} have no feasible solution, that is, b ∈ C := {Ax : x ≥ 0}. Then, by the separating hyperplane theorem, there is y such that y • b > sup y • c c∈C or T y • b > sup y • (Ax)=supA y • x. (1) x≥0 x≥0

Since 0 ∈ C we have y • b > 0. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 27

T T Furthermore, A y ≤ 0. Since otherwise, say (A y)1 > 0, one can have a vector x¯ ≥ 0 such that x¯1 = α>0, x¯2 = ... =¯xn =0, from which

T T T sup A y • x ≥ A y • x¯ =(A y)1 · α x≥0 and it tends to ∞ as α →∞. This is a contradiction because T supx≥0 A y • x is bounded from above by (1). Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 28

Farkas’ Lemma variant

m×n n T Theorem 6 Let A ∈R and c ∈R . Then, the system {y : A y ≤ c} T has a solution y if and only if that Ax = 0, x ≥ 0, c x < 0 has no feasible solution x.

T Again, a vector x ≥ 0, with Ax = 0 and c x < 0, is called a infeasibility T certificate for the system {y : A y ≤ c}.

Example: Let A =(1;−1) and c =(1;−2). Then, x = (1; 1) is an T infeasibility certificate for {y : A y ≤ c}. Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 29

Alternative Systems for General Cone?

{x : Ax = b, x ∈ C} and {y : −AT y ∈ C∗, bT y > 0}?

Counterexample: ⎛ ⎞ ⎛ ⎞ 10 01 ⎝ ⎠ ⎝ ⎠ A1 = ,A2 = 00 10 and ⎛ ⎞ 0 ⎝ ⎠ 2 b = ,C= S+. 2 Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 30

Farkas Lemma for General Convex Cone

Theorem 7 Consider system {x : Ax = b, x ∈ K} for a (closed) convex T ∗ cone K. Suppose that there exists vector y¯ such that −A y¯ ∈ int K . Then,

• Set C := {Ax : x ∈ K} is a closed convex set.

• The system {x : Ax = b, x ∈ K} has a feasible solution x if and only if T ∗ T T that {y : −A y ∈ K , b y > 0, (b y =1)} has no feasible solution. T Corollary 1 Consider system {(y, s): A y + s = c, s ∈ K} for a (closed) convex cone K. Suppose that there exists vector x¯ such that ∗ Ax¯ = 0, x¯ ∈ int K . Then, T • Set C := {A y + s : s ∈ K} is a closed convex set. ∗ • The system {x : Ax = 0, c • x = −1, x ∈ K } has a feasible solution T x if and only if that {(y, s): A y + s = c, s ∈ K} has no feasible solution.