Lecture 2 Topology of Sets in Rn January 13, 2016 This material was jointly developed with Angelia Nedi´cat UIUC for IE 598ns Uday V. Shanbhag Lecture 2 Outline • Vectors, Matrices, Norms, Convergence • Open and Closed Sets • Special Sets: Subspace, Affine Set, Cone, Convex Set • Special Convex Sets: Hyperplane, Polyhedral Set, Ball • Special Cones: Normal Cone, Dual Cone, Tangent Cone • Some Operations with Sets: • Set Closure, Interior, Boundary • Convex Hull, Conical Hull, Affine Hull • Relative Closure, Interior, Boundary • Functions • Special Functions: Linear, Quadratic Stochastic Optimization 1 Uday V. Shanbhag Lecture 2 Notation: Vectors and Matrices We consider the space of n-dimensional real vectors denoted by Rn 2 3 x1 n 6 . 7 • Vector x 2 is viewed as a column: x = 6 . 7 R 4 5 xn 0 • The prime denotes a transpose: x = [x1; : : : ; xn] • For a matrix A, we use aij or [A]ij to denote its (i; j)-th entry • The matrix A is positive semidefinite (A ≥ 0) when 0 n x Ax ≥ 0 for all x 2 R • The matrix A is positive definite (A > 0) when 0 n x Ax > 0 for all x 2 R ; x 6= 0 Stochastic Optimization 2 Uday V. Shanbhag Lecture 2 Inner Product and Norms • For vectors x 2 Rn and y 2 Rn, the inner product is n 0 X x y = xiyi i=1 • Norm (length) of a vector: p q 0 Pn 2 • Euclidean norm kxk = x x = i=1 xi (l2-norm) Pn • Sum-norm kxk1 = i=1 jxij (l1-norm) • Max-norm kxk1 = max1≤i≤n jxij (l1-norm) • Matrix norm induced by (Euclidean) vector norm kAxk kAk = sup x6=0 kxk kAxk = sup kxk=1 kxk Stochastic Optimization 3 Uday V. Shanbhag Lecture 2 Norm Properties, Distance, Orthogonal Vectors • Properties of a norm • Nonnegativity: kxk ≥ 0 for every x 2 Rn • Uniqueness of the zero-value: kxk = 0 if and only if x = 0 • Homogeneity: kλxk = jλj kxk for any scalar λ and any x 2 Rn • Triangle relation: kx + yk ≤ kxk + kyk for every x and y 2 Rn • (Euclidean) Distance between two vectors x 2 Rn and y 2 Rn v u n uX 2 kx − yk = t (xi − yi) i=1 • Vectors x and y are orthogonal when x0y = 0 • For orthogonal vectors x and y: Pythagorean Theorem kx + yk2 = kxk2 + kyk2 • For any vectors x and y: Schwartz Inequality: jx0yj ≤ kxk kyk Stochastic Optimization 4 Uday V. Shanbhag Lecture 2 Vector Sequence n Given a vector sequence fxkg ⊂ R , we say that • The sequence is bounded when for some scalar C, we have kxkk ≤ C for all k • The sequence converges to a vector x~ 2 Rn when limk!1 kxk − x~k = 0 (xk ! x~) • Vector y is an accumulation point of the sequence when there is a subsequence fxkig such that xki ! y as i ! 1 k k • Example: xk = (1 + (−1) a) with 0 < a < 1. Accum. points? • Bolzano Theorem n A bounded sequence fxkg ⊂ R has at least one limit point Stochastic Optimization 5 Uday V. Shanbhag Lecture 2 Set Topology Typically denoted by capital letters C, X, Y , Z, etc. Let X be a set in Rn. • A vector x is an accumulation (or a limit) point of the set X when there is a sequence fxkg ⊆ X such that xk ! x • The set X is closed when it contains all of its accumulation points • The set X is open when its complement set Xc = fx 2 Rn j x 62 Xg is closed • Rn and ; are the only sets that are both open and closed • The set X is bounded when for some C: kxk ≤ C for all x 2 X • (Th.) The set X is compact when it is both closed and bounded n Let fXi j i 2 Ig be a family of closed sets Xi ⊆ R • The intersection \i2IXi is closed • When the set I is finite (I = f1; : : : ; mg) the union [i2IXi is closed Stochastic Optimization 6 Uday V. Shanbhag Lecture 2 Special Sets Let X be a set in Rn. We say that X is: • subspace when αx + βy 2 X for every x; y 2 X and any α, β 2 R 0 n • Example: fx j x e = 0g where e 2 R with ei = 1 for all i • Is a subspace a closed set? Bounded? • affine when it is a translated subspace: X = x + S for an x 2 X and a subspace S • cone when λx 2 X for every λ ≥ 0 and every x 2 X • convex when αx + (1 − α)y 2 X for any x; y 2 X and α 2 [0; 1] Stochastic Optimization 7 Uday V. Shanbhag Lecture 2 Convex Sets Is a subspace convex set? Affine set? Special convex sets: n • (Euclidean) Ball centered at x0 2 R with a radius r > 0 n fx 2 R j kx − x0k ≤ rg (closed ball) n fx 2 R j kx − x0k < rg (open ball) • A closed ball is convex; An open ball is convex Stochastic Optimization 8 Uday V. Shanbhag Lecture 2 More Convex Sets • Half-space with normal vector a 6= 0: fx j a0x ≤ bg • Hyperplane with normal vector a 6= 0: fx j a0x = bg (b is a scalar) • Polyhedral set (given by finitely many linear inequalities): 0 fx j aix ≤ bi; i = 1; : : : ; mg = fx j Ax ≤ bg A is an m × n matrix and b 2 Rm Stochastic Optimization 9 Uday V. Shanbhag Lecture 2 Dual Cone Let K ⊆ Rn be a cone. Def. A dual cone of K is the following set ∗ n 0 K = fd 2 R j d y ≥ 0 for all y 2 Kg • Dual cone appears in formulations of complementarity problems • Dual cone is always closed and convex (regardless of K) Stochastic Optimization 10 Uday V. Shanbhag Lecture 2 Normal Cone of a Set Let X ⊆ Rn be a nonempty set, and let x^ 2 X. Def. A normal cone of the set X at the point x^ is the following set N(^x; X) = fy 2 Rn j y0(x − x^) ≤ 0 for all x 2 Xg Vectors in this set are called normal vectors to the set X at x^ X ~x N(~x ; X ) = {0} ^x N(^x ; X ) • Normal cone plays an important role in optimality conditions • Normal cone is always closed and convex (regardless of X) Stochastic Optimization 11 Uday V. Shanbhag Lecture 2 Tangent Cone of a Set Let X ⊆ Rn be a nonempty set, and let x^ 2 X. Def. A tangent cone of the set X at the point x^ is the following set n xk − x^ T (^x; X) = y 2 R j y = lim with fxkg ⊂ X; fλkg ⊂ R++ k!1 λk s. t. xk ! x;^ λk ! 0g Vectors in this set are called tangent vectors to the set X at x^ Prop. The tangent cone T (^x; X) is equivalently given by ( n xk − x^ T (^x; X) = βy 2 R j y = lim with fxkg ⊂ X; k!1 kxk − x^k xk ! x;^ xk 6=x; ^ β ≥ 0g Stochastic Optimization 12 Uday V. Shanbhag Lecture 2 Tangent Cone Illustration T(^x ; X ) X ~x ^x Tangent cone T (^x; X) for any set X and any x^ 2 X: • plays an important role in optimality conditions • always closed regardless of the properties of X • when X is convex, then T (^x; X) is convex Stochastic Optimization 13 Uday V. Shanbhag Lecture 2 Tangent and Normal Cone Let X ⊂ Rn and let x^ 2 X. Then, we have −N(^x; X) ⊆ T (^x; X)∗ X T(^x ; X )* ~x ^x N(^x ; X ) Prop. If X is convex, then T (^x; X)∗ = −N(^x; X) for all x^ 2 X Stochastic Optimization 14 Uday V. Shanbhag Lecture 2 Set Interior, Closure and Boundary Let X ⊆ Rn. • A vector x^ 2 X is an interior point of X if there exist a ball B(^x; r) such that B(^x; r) ⊂ X • The interior of the set X is the set of all interior points of X; it is denoted by int(X) • A vector x^ 2 X is a accumulation point of X if there exist a sequence fxkg ⊂ X converging to x^ • The closure of the set X is the set of all accumulation points of X; it is denoted by cl(X) • The boundary of the set X is the intersection of the closure set of X and the closure set of its complement (Rn n X), i.e., bd(X) = cl(X) \ cl(Rn n X) 2 Example: X = f(x1; x2) 2 R j 0 ≤ x1 ≤ 1; 0 < x2 < 1g Stochastic Optimization 15 Uday V. Shanbhag Lecture 2 Convex and Conical Hulls A convex combination of vectors x1; : : : ; xm is a vector of the form Pm α1x1 + ::: + αmxm αi ≥ 0 for all i and i=1 αi = 1 The convex hull of a set X is the set of all convex combinations of the vectors in X, denoted conv(X) A conical combination of vectors x1; : : : ; xm is a vector of the form λ1x1 + ::: + λmxm with λi ≥ 0 for all i The conical hull of a set X is the set of all conical combinations of the vectors in X, denoted by cone(X) Stochastic Optimization 16 Uday V. Shanbhag Lecture 2 Affine Hull An affine combination of vectors x1; : : : ; xm is a vector of the form Pm t1x1 + ::: + tmxm with i=1 ti = 1, ti 2 R for all i The affine hull of a set X is the set of all affine combinations of the vectors in X, denoted aff(X) The dimension of a set X is the dimension of the affine hull of X dim(X) = dim(aff(X)) Stochastic Optimization 17 Uday V.