Lecture 2

Topology of Sets in Rn

January 13, 2016

This material was jointly developed with Angelia Nedi´cat UIUC for IE 598ns Uday V. Shanbhag Lecture 2 Outline • Vectors, Matrices, Norms, Convergence • Open and Closed Sets • Special Sets: Subspace, Affine Set, Cone, Convex Set • Special Convex Sets: Hyperplane, Polyhedral Set, Ball • Special Cones: Normal Cone, Dual Cone, Tangent Cone • Some Operations with Sets: • Set Closure, Interior, Boundary • Convex Hull, Conical Hull, Affine Hull • Relative Closure, Interior, Boundary • Functions • Special Functions: Linear, Quadratic

Stochastic Optimization 1 Uday V. Shanbhag Lecture 2 Notation: Vectors and Matrices

We consider the space of n-dimensional real vectors denoted by Rn   x1 n  .  • Vector x ∈ is viewed as a column: x =  .  R   xn 0 • The prime denotes a transpose: x = [x1, . . . , xn]

• For a matrix A, we use aij or [A]ij to denote its (i, j)-th entry • The matrix A is positive semidefinite (A ≥ 0) when

0 n x Ax ≥ 0 for all x ∈ R

• The matrix A is positive definite (A > 0) when

0 n x Ax > 0 for all x ∈ R , x 6= 0

Stochastic Optimization 2 Uday V. Shanbhag Lecture 2 Inner Product and Norms

• For vectors x ∈ Rn and y ∈ Rn, the inner product is n 0 X x y = xiyi i=1 • Norm (length) of a vector: √ q 0 Pn 2 • Euclidean norm kxk = x x = i=1 xi (l2-norm) Pn • Sum-norm kxk1 = i=1 |xi| (l1-norm)

• Max-norm kxk∞ = max1≤i≤n |xi| (l∞-norm) • Matrix norm induced by (Euclidean) vector norm

kAxk kAk = sup x6=0 kxk kAxk = sup kxk=1 kxk

Stochastic Optimization 3 Uday V. Shanbhag Lecture 2 Norm Properties, Distance, Orthogonal Vectors

• Properties of a norm • Nonnegativity: kxk ≥ 0 for every x ∈ Rn • Uniqueness of the zero-value: kxk = 0 if and only if x = 0 • Homogeneity: kλxk = |λ| kxk for any scalar λ and any x ∈ Rn • Triangle relation: kx + yk ≤ kxk + kyk for every x and y ∈ Rn • (Euclidean) Distance between two vectors x ∈ Rn and y ∈ Rn v u n uX 2 kx − yk = t (xi − yi) i=1 • Vectors x and y are orthogonal when x0y = 0 • For orthogonal vectors x and y: Pythagorean Theorem kx + yk2 = kxk2 + kyk2 • For any vectors x and y: Schwartz Inequality: |x0y| ≤ kxk kyk

Stochastic Optimization 4 Uday V. Shanbhag Lecture 2 Vector Sequence

n Given a vector sequence {xk} ⊂ R , we say that • The sequence is bounded when for some scalar C, we have

kxkk ≤ C for all k

• The sequence converges to a vector x˜ ∈ Rn when limk→∞ kxk − x˜k = 0 (xk → x˜)

• Vector y is an accumulation point of the sequence when there is a

subsequence {xki} such that xki → y as i → ∞

k k • Example: xk = (1 + (−1) a) with 0 < a < 1. Accum. points?

• Bolzano Theorem n A bounded sequence {xk} ⊂ R has at least one limit point

Stochastic Optimization 5 Uday V. Shanbhag Lecture 2 Set Topology Typically denoted by capital letters C, X, Y , Z, etc. Let X be a set in Rn. • A vector x is an accumulation (or a limit) point of the set X when

there is a sequence {xk} ⊆ X such that xk → x • The set X is closed when it contains all of its accumulation points • The set X is open when its complement set Xc = {x ∈ Rn | x 6∈ X} is closed • Rn and ∅ are the only sets that are both open and closed • The set X is bounded when for some C: kxk ≤ C for all x ∈ X • (Th.) The set X is compact when it is both closed and bounded n Let {Xi | i ∈ I} be a family of closed sets Xi ⊆ R

• The intersection ∩i∈IXi is closed

• When the set I is finite (I = {1, . . . , m}) the union ∪i∈IXi is closed

Stochastic Optimization 6 Uday V. Shanbhag Lecture 2 Special Sets Let X be a set in Rn. We say that X is: • subspace when αx + βy ∈ X for every x, y ∈ X and any α, β ∈ R 0 n • Example: {x | x e = 0} where e ∈ R with ei = 1 for all i • Is a subspace a closed set? Bounded?

• affine when it is a translated subspace: X = x + S for an x ∈ X and a subspace S

• cone when λx ∈ X for every λ ≥ 0 and every x ∈ X

• convex when αx + (1 − α)y ∈ X for any x, y ∈ X and α ∈ [0, 1]

Stochastic Optimization 7 Uday V. Shanbhag Lecture 2 Convex Sets

Is a subspace convex set? Affine set? Special convex sets:

n • (Euclidean) Ball centered at x0 ∈ R with a radius r > 0

n {x ∈ R | kx − x0k ≤ r} (closed ball)

n {x ∈ R | kx − x0k < r} (open ball) • A closed ball is convex; An open ball is convex

Stochastic Optimization 8 Uday V. Shanbhag Lecture 2 More Convex Sets • Half-space with normal vector a 6= 0: {x | a0x ≤ b} • Hyperplane with normal vector a 6= 0: {x | a0x = b} (b is a scalar)

• Polyhedral set (given by finitely many linear inequalities): 0 {x | aix ≤ bi, i = 1, . . . , m} = {x | Ax ≤ b} A is an m × n matrix and b ∈ Rm

Stochastic Optimization 9 Uday V. Shanbhag Lecture 2 Dual Cone Let K ⊆ Rn be a cone. Def. A dual cone of K is the following set

∗ n 0 K = {d ∈ R | d y ≥ 0 for all y ∈ K}

• Dual cone appears in formulations of complementarity problems • Dual cone is always closed and convex (regardless of K)

Stochastic Optimization 10 Uday V. Shanbhag Lecture 2 Normal Cone of a Set Let X ⊆ Rn be a nonempty set, and let xˆ ∈ X. Def. A normal cone of the set X at the point xˆ is the following set N(ˆx; X) = {y ∈ Rn | y0(x − xˆ) ≤ 0 for all x ∈ X} Vectors in this set are called normal vectors to the set X at xˆ

X ~x N(~x ; X ) = {0}

^x

N(^x ; X )

• Normal cone plays an important role in optimality conditions • Normal cone is always closed and convex (regardless of X)

Stochastic Optimization 11 Uday V. Shanbhag Lecture 2 Tangent Cone of a Set

Let X ⊆ Rn be a nonempty set, and let xˆ ∈ X. Def. A tangent cone of the set X at the point xˆ is the following set

 n xk − xˆ T (ˆx; X) = y ∈ R | y = lim with {xk} ⊂ X, {λk} ⊂ R++ k→∞ λk

s. t. xk → x,ˆ λk → 0}

Vectors in this set are called tangent vectors to the set X at xˆ Prop. The tangent cone T (ˆx; X) is equivalently given by

( n xk − xˆ T (ˆx; X) = βy ∈ R | y = lim with {xk} ⊂ X, k→∞ kxk − xˆk

xk → x,ˆ xk 6=x, ˆ β ≥ 0}

Stochastic Optimization 12 Uday V. Shanbhag Lecture 2 Tangent Cone Illustration

T(^x ; X )

X

~x

^x

Tangent cone T (ˆx; X) for any set X and any xˆ ∈ X:

• plays an important role in optimality conditions • always closed regardless of the properties of X • when X is convex, then T (ˆx; X) is convex

Stochastic Optimization 13 Uday V. Shanbhag Lecture 2 Tangent and Normal Cone

Let X ⊂ Rn and let xˆ ∈ X. Then, we have −N(ˆx; X) ⊆ T (ˆx; X)∗

X

T(^x ; X )* ~x

^x N(^x ; X )

Prop. If X is convex, then

T (ˆx; X)∗ = −N(ˆx; X) for all xˆ ∈ X

Stochastic Optimization 14 Uday V. Shanbhag Lecture 2 Set Interior, Closure and Boundary

Let X ⊆ Rn. • A vector xˆ ∈ X is an interior point of X if there exist a ball B(ˆx, r) such that B(ˆx, r) ⊂ X • The interior of the set X is the set of all interior points of X; it is denoted by int(X) • A vector xˆ ∈ X is a accumulation point of X if there exist a sequence

{xk} ⊂ X converging to xˆ • The closure of the set X is the set of all accumulation points of X; it is denoted by cl(X) • The boundary of the set X is the intersection of the closure set of X and the closure set of its complement (Rn \ X), i.e., bd(X) = cl(X) ∩ cl(Rn \ X) 2 Example: X = {(x1, x2) ∈ R | 0 ≤ x1 ≤ 1, 0 < x2 < 1}

Stochastic Optimization 15 Uday V. Shanbhag Lecture 2 Convex and Conical Hulls

A convex combination of vectors x1, . . . , xm is a vector of the form Pm α1x1 + ... + αmxm αi ≥ 0 for all i and i=1 αi = 1 The convex hull of a set X is the set of all convex combinations of the vectors in X, denoted conv(X)

A conical combination of vectors x1, . . . , xm is a vector of the form

λ1x1 + ... + λmxm with λi ≥ 0 for all i The conical hull of a set X is the set of all conical combinations of the vectors in X, denoted by cone(X)

Stochastic Optimization 16 Uday V. Shanbhag Lecture 2 Affine Hull

An affine combination of vectors x1, . . . , xm is a vector of the form Pm t1x1 + ... + tmxm with i=1 ti = 1, ti ∈ R for all i The affine hull of a set X is the set of all affine combinations of the vectors in X, denoted aff(X)

The dimension of a set X is the dimension of the affine hull of X dim(X) = dim(aff(X))

Stochastic Optimization 17 Uday V. Shanbhag Lecture 2 Relative Interior, Closure, and Boundary

Let X ⊆ Rn. • A vector xˆ ∈ X is a relative interior point of X if there exist a ball B(ˆx, r) such that B(ˆx, r) ∩ aff(X) ⊂ X

• The relative interior of the set X is the set of all interior points of X; it is denoted by rint(X)

• The relative boundary of the set X is the intersection of the affine hull aff(X) and the boundary bd(X)

2 Example: X = {(x1, x2) ∈ R | 0 ≤ x1 ≤ 1, x2 = 1} aff(X) =?

Stochastic Optimization 18 Uday V. Shanbhag Lecture 2 Mappings

Let X ⊆ Rn. • We write F : X → Rm to denote a mapping F from the set X to Rm • F is a function when F : X → R (m = 1) • We refer to X as a domain of the mapping F • The image of X under mapping F is the following set

m F (X) = {y ∈ R | y = F (x) for some x ∈ X}

• The inverse image of Y ⊆ Rm under mapping F is the following set

F −1(Y ) = {x ∈ X | F (x) ∈ Y }

Stochastic Optimization 19 Uday V. Shanbhag Lecture 2 Special Mappings and Functions A mapping F : X → Rm with X ⊆ Rn is • affine when for all x ∈ X,

F (x) = Ax + b for some matrix A and a vector b ∈ Rm

n Example: Space translation (by a given x0 ∈ R ) is an affine mapping n n n from R to R : F (x) = x + x0 for all x ∈ R • linear when for all x ∈ X, we have F (x) = Ax for a matrix A

n Example: For a coordinate subspace S = {x ∈ R | xi = 0, for i ∈ I}, where I is a subset of coordinate indices (I ⊆ {1, . . . , n}, the projection on S is a linear mapping from Rn to Rn:

F (x) = P x with [P ]ii = 1 for i 6∈ I and Pij = 0 otherwise

Stochastic Optimization 20 Uday V. Shanbhag Lecture 2 Special Functions

A function f : Rn → R is • quadratic when for all x f(x) = x0Qx + a0x + b for a matrix Q, a ∈ Rn and b ∈ R

• affine when for all x f(x) = a0x + b for a vector a ∈ Rn and b ∈ R

• linear when for all x f(x) = a0x for a vector a ∈ Rn

Stochastic Optimization 21 Uday V. Shanbhag Lecture 2 References for this lecture The material for this lecture:

• (B) Bertsekas D.P. Nonlinear Programming

• See Appendix there for topology in Rn and linear algebra

• (BNO) Bertsekas, Nedi´c,Ozdaglar Convex Analysis and Optimization Chapter 1 for convex sets and functions

• (FP) Facchinei and Pang Finite Dimensional ..., Vol I • Chapter 1 for Normal Cone, Dual Cone, and Tangent Cone

• NOTE!!! • Normal cone definition used in (FP) is different than that of (B,BNO)

Stochastic Optimization 22