LCoP Part II – Preliminaries and Convex Cone Structures Shu-Cherng Fang Department of Industrial and Systems Engineering Graduate Program in Operations Research North Carolina State University December 2017 Conic Programming 1 / 38 Preliminaries and Convex Cone Structures Content • Vectors, Matrices, and Spaces • Inner Products and Norms • Open, Closed, Interior, and Boundary Sets • Functions • Linear Systems • Convex Sets and Functions Conic Programming 2 / 38 Vectors, Matrices and Spaces • Real numbers: R, R+, R++ • Euclidean space: Rn Rn • First orthant: + • n-dimensional (column) vector: T x = (x1; x2; : : : ; xn) • Matrices space: Rm×n m×n • Matrix: M 2 R , ith row Mi·, jth column M·j, ijth entry Mij(Mi;j) • Symmetric square matrices space (n(n + 1)=2-dimensional space): Sn = fM 2 Rn×n j M = M T g: Conic Programming 3 / 38 Vectors, Matrices and Spaces Given M 2 Rm×n, N 2 Rn×m, S 2 Rn×n • Determinant: det(S) • Trace: tr(S) tr(MN) = tr(NM) • Null space: N (M)= fx 2 RnjMx = 0g: • Range space: R(M)= fy 2 Rmjy = Mx for some x 2 Rng: • Positive semidefinite matrix: S 0 () zT Sz ≥ 0; 8 z 2 Rn • Positive definite matrix: S 0 () zT Sz > 0; 8 z 2 Rn and z 6= 0 Conic Programming 4 / 38 Vectors, Matrices and Spaces Theorem: (Schur complementary theorem) Given AB X = and S = C − BT A−1B; BT C if A 0 then X ()0 , S ()0 Conic Programming 5 / 38 Inner Products and Norms • Inner products: T P x • y = x y = i xiyi T P X • Y = tr(X Y ) = i;j XijYij • Norms: p • Euclidean norm: kxk2 = x • x Pn p 1=p • p-norm: kxkp = ( i=1 jxij ) for p ≥ 1 • Infinity-norm: kxk1 = maxfjx1j;:::; jxnjg • Frobenius norm: p p T kXkF = X • X = tr(X X) • Note that: xT Ax = A • xxT Conic Programming 6 / 38 Open, Closed, Interior and Boundary Sets • Neighborhood: N(x0; ) = fx 2 Rnj kx − x0k < g: • Open: X ⊂ Rn is open if for any x 2 X , there exists > 0 such that N(x; ) ⊂ X . • Closed: X ⊂ Rn is closed, if RnnX = fx 2 Rnjx2 = X g is open. • Closure of a set X ⊂ Rn is the smallest closed set containing X and is denoted as cl(X ). Conic Programming 7 / 38 Open, Closed, Interior and Boundary Sets • Interior: the interior of a given set X ⊂ Rn is int(X ) = fx 2 X j9 x > 0 such that N(x; x) ⊂ X g • Boundary of a set X ⊂ Rn: bdry(X )= cl(X )nint(X ) = fx 2 cl(X )jx2 = int(X )g • Bounded: a set X ⊂ Rn is bounded if there exist an r > 0 such that kxk < r; 8x 2 X Conic Programming 8 / 38 Functions • Continuous: f : X ⊂ Rn is continuous at x0 if ( i ) x0 2 X 0 (i i) limx!x0 f(x) = f(x ) • Continuous function: f 2 C0(X ) means f is continuous at all points in X ⊂ Rn. • Gradient: For f : X ⊂ Rn ! R @f(x) @f(x) rf(x) = [ ; ··· ; ]1×n @x1 @xn • Hessian: For f : X ⊂ Rn ! R @2f(x) F (x) = [ ]n×n @xi@xj • Continuously differentiable function: f 2 Cp(X ) (p = 1; 2; ··· ) means f is p-th continuously differentiable over X ⊂ Rn. Conic Programming 9 / 38 Functions Theorem (Taylor theorem) Let X be open, f 2 Cp(X ), x1; x2 2 X , x1 6= x2 and x(θ) = θx1 + (1 − θ)x2 2 X ; 8 0 ≤ θ ≤ 1: Then 9 x¯ = θx¯ 1 + (1 − θ¯)x2 2 X , 0 < θ¯ < 1, s.t. p−1 X 1 1 f(x2) = f(x1) + dkf(x1; x2 − x1) + dpf(¯x; x2 − x1) k! p! k=1 where dkf(x; h) is the k-th order differential of function f along h. Conic Programming 10 / 38 Functions: Big O and Small o Let g(·) be a real-valued function on R. • g(x) = O(m(x)) 9 c ≥ 0 such that g(x) ≤ c as x ! 0 (or + 1) m(x) • g(x) = o(m(x)) g(x) = 0 as x ! 0 (or + 1) m(x) Conic Programming 11 / 38 Functions Taylor theorem in small o formulation: • p = 1 f(x + h) = f(x) + rf(x)h + o(khk) • p = 2 1 f(x + h) = f(x) + rf(x)h + hT F (x)h + o(khk2) 2 Conic Programming 12 / 38 Linear Systems Given x1; ··· ; xm 2 Rn • Linear combination: m X i λix ; i=1 where λi 2 R, i = 1; : : : ; m. • Linearly independent m X i λix = 0 ) λ1 = ··· = λm = 0 i=1 • Affine combination: a linear combination with m X λi = 1 i=1 • Affinely independent: if x2 − x1, ··· , xm − x1 are linearly independent. Conic Programming 13 / 38 Linear Systems • Convex combination: a linear combination with m X λi = 1 and λi ≥ 0; i = 1; : : : ; m i=1 • Hyperplane: n n T X X = fx 2 R ja x = aixi = bg i=1 • Affine space: affine combination of any two points in the space is still in the space. (An intersection of finitely many hyperplanes.) • Linear subspace: an affine space containing the origin. We can always transform an affine space Y ⊂ Rn into a linear subspace X ⊂ Rn by choosing x0 2 Y such that X = fx − x0jx 2 Yg Conic Programming 14 / 38 Linear Systems • Half space: n n T X X = fx 2 R ja x = aixi ≤ bg i=1 • Polyhedron: an intersection of finitely many half spaces. • Polytope: a bounded polyhedron • Dimension of a linear subspace: the maximum number of linearly independent vectors in the subspace. • Dimension of an affine space: the dimension of the transformed linear subspace. • Dimension of a polyhedron: the dimension of the smallest affine space containing it. Conic Programming 15 / 38 Linear Systems • Linear equations 1 a • x = b1 a2 • x = b 2 ) Ax = b; ········· m a • x = bm where a1; ··· ; am and x are all in Rn. A1 • X = b1 A • X = b 2 2 )A X = b; ········· Am • X = bm n where A1; ··· ;Am and X are all in S . ∗ Pm • For convenience, A y = i=1 yiAi. Conic Programming 16 / 38 Convex Sets and Properties • A set X ⊂ Rn is convex if for any x1 2 X and x2 2 X , we have λx1 + (1 − λ)x2 2 X , for all 0 ≤ λ ≤ 1. • Convex hull: the smallest convex set containing a given set Rn Pm i N conv(X )= fx 2 jx = i=1 λiy for some m 2 +; Pm i λi ≥ 0; i=1 λi = 1; and y 2 X ; i = 1; : : : ; mg • Dimension of a convex set: the dimension of the smallest affine space containing it. • Relative interior of a convex set X ⊂ Rn: suppose H is the smallest affine space containing X , ri(X )= fx 2 Rnj9 open set Y ⊆ Rn such that x 2 Y \ H ⊂ X g • Supporting hyperplane H = fx 2 RnjaT x = bg of a convex set X : aT y ≥ b; 8 y 2 X and X \ H 6= ;: Conic Programming 17 / 38 Convex Functions and Properties • Epigraph of a function f : X ⊂ Rn ! R epif = f(x; λ) 2 Rn+1jλ ≥ f(x); x 2 X g • Closed function: if epif is a closed set. • Convex function: if epif is a convex set. • Concave function: if −f is a convex function. • Convex hull function conv(f) of a function f : X ⊂ Rn ! R is a function on X such that epi(conv(f)) = conv(epi(f)): Lemma f : X ⊂ Rn ! R is a convex function if and only if for any x1, x2 2 X and 0 ≤ λ ≤ 1, we have f(λx1 + (1 − λ)x2) ≤ λf(x1) + (1 − λ)f(x2): Conic Programming 18 / 38 Convex Functions and Properties • Subgradient d 2 Rn of a convex function f : X ⊂ Rn at x 2 X : if for any y 2 X , f(y) ≥ f(x) + dT (y − x) • The set f(y; λ) 2 Rn+1jλ − dT y = f(x) − dT xg is a supporting hyperplane of epif at x. • Subdifferential of a convex function f : X ⊂ Rn at x 2 X : @f(x) = fd 2 Rnjd is a subgradient of f at xg Conic Programming 19 / 38 Convex Functions and Properties Figure: (x; f(x)) $ (m; b) or (y; h(y)) Conic Programming 20 / 38 Convex Functions and Properties • Conjugate (transform) of f : X ⊂ Rn ! R: h(y) = supfy • x − f(x)g x2X with h being defined on Y = fy 2 Rnjh(y) < +1g. Lemma h : Y is a closed, convex function. Lemma (Fenchel’s inequality) Given f : X and its conjugate h : Y, then x • y ≤ f(x) + h(y); 8 x 2 X and y 2 Y: Moreover, x • y = f(x) + h(y) () y 2 @f(x) Conic Programming 21 / 38 Conjugate Functions and Properties Let f : X ⊂ Rn ! R be a function with its conjugate transform h : Y. • For α 2 R, the conjugate of f + α is h − α. • For a 2 Rn, the conjugate of f~(x) = f(x) + x • a on X is h~(y) = h(y − a), 8 y 2 Y. • For a 2 Rn, the conjugate of f¯(x) = f(x − a) on X is h¯(y) = h(y) + y • a, 8 y 2 Y. • y For λ > 0, the conjugate of f1(x) = λf(x) on X is h1(y) = λh( λ ), 8 y 2 λY. • x For λ > 0, the conjugate of f2(x) = f( λ ) on λX is h2(y) = h(λy), 8 y 2 Y/λ. Theorem Assume that f1 : X and f2 : X have the same convex hull function.