Lecture 4 Closed Functions September 4, 2007 Lecture 4 Outline • Log-Transformation • Jensen’s Inequality • Level Sets • Closed Functions • Convexity and Continuity Convex Optimization 1 Lecture 4 Log-Transformation of Variables Useful for transforming a nonconvex function to a convex one A posynomial is a function of y1 > 0, . , yn > 0 of the form a1 an g(y1, . , yn) = by1 ··· yn with scalars b > 0 and ai > 0 for all i. • A posynomial need not be convex • Log-transformation of variables xi = ln yi for all i • We have a convex function 0 f(x) = bea1x1 ··· eanxn = bea x Convex Optimization 2 Lecture 4 Log-Transformation of Functions Replacing f with ln f [when f(x) > 0 over domf] Useful for: • Transforming non-separable functions to separable ones n 1/n Example: (Geometric Mean) f(x) = (Πi=1xi) for x with xi > 0 for all i is non-separable. Using F (x) = ln f(x), we obtain a separable F , n 1 X F (x) = ln xi n i=1 • Separable structure of objective function is advantageous in distributed optimization Convex Optimization 3 Lecture 4 General Convex Inequality Basic convex inequality: For a convex f, we have for x, y ∈ domf and α ∈ (0, 1), f(αx + (1 − α)y) ≤ αf(x) + (1 − α)f(y) General convex inequality: For a convex f and any convex combination of the points in domf, we have m ! m X X f αixi ≤ αif(xi) i=1 i=1 Pm (xi ∈ domf and αi > 0 for all i, i=1 αi = 1, m > 0 integer) Pm A convex combination i=1 αixi can be viewed as the expectation of a random vector z having outcomes z = xi with probability αi Convex Optimization 4 Lecture 4 Jensen’s Inequality General convex inequality can be interpreted as: • For a convex f and a (finite) discrete random variable z with outcomes zi ∈ domf, we have f(Ez) ≤ E [f(z)] General Jensen’s inequality: The above relation holds for a convex f and, a random variable z with outcomes in domf and a finite expectation Ez Convex Optimization 5 Lecture 4 Level Sets Def. Given a scalar c ∈ R and a function f : Rn → R, a (lower) level set of f associated with c is given by Lc(f) = {x ∈ domf | f(x) ≤ c} 2 n x Examples: f(x) = kxk for x ∈ R , f(x1, x2) = e 1 • Every level set of a convex function is convex • Converse is false: Consider f(x) = −ex for x ∈ R Def. A function g is concave when −g is convex • Every (upper) level set of a concave function is convex Convex Optimization 6 Lecture 4 Convex Extended-Value Functions • The definition of convexity that we have used thus far is applicable to functions mapping from a subset of Rn to Rn. It does not apply to extended-value functions mapping from a subset of Rn to the extended set R ∪ {−∞, +∞}. • The general definition of convexity relies on the epigraph of a function • Let f be a function taking values in R ∪ {−∞, +∞}. The epigraph of f is the set given by n n epif = {(x, w) ∈ R × R | f(x) ≤ w for some x ∈ R }. • Def. The extended-value function f is convex when its epigraph epif is convex set (in Rn × R). • This definition coincides with the one we have used for a function f with values in R. Convex Optimization 7 Lecture 4 Closed Functions Def. A function f is closed if its epigraph epif is a closed set in Rn × R, i.e., for every sequence {(xk, wk)} ⊂ epif converging to some (ˆx, wˆ), we have (ˆx, wˆ) ∈ epif Examples Affine functions are closed [f(x) = a0x + b] Quadratic functions are closed [f(x) = x0P x + a0x + b] Continuous functions are closed • A class of closed functions is larger than the class of continuous functions • For example f(x) = 0 for x = 0, f(x) = 1 for x > 0, and f(x) = +∞ otherwise This f is closed but not continuous Convex Optimization 8 Lecture 4 Closed Function Properties Lower-Semicontinuity Def. A function f is lower-semicontinuous at a given vector x0 if for every sequence {xk} converging to x0, we have f(x0) ≤ lim infk→0 f(xk) We say that f is lower-semicontinuous over a set X if f is lower- semicontinuous at every x ∈ X Th. For a function f : Rn → R ∪ {−∞, +∞} the following statements are equivalent (i) f is closed (ii) Every level set of f is closed (iii) f is lower-semicontinuous (l.s.c.) over Rn Convex Optimization 9 Lecture 4 Proof of Theorem (i)⇒(ii) Let c be any scalar and consider Lc(f). If Lc(f) = ∅, then Lc(f) is closed. Suppose now that Lc(f) 6= ∅. Pick {xk} ⊂ Lc(f) such that n xk → x¯ for some x¯ ∈ R . We have f(xk) ≤ c for all k, implying that (xk, c) ∈ epif for all k. Since (xk, c) → (¯x, c) and epif is closed, it follows that (¯x, c) ∈ epif. Consequently f(¯x) ≤ c, showing that x¯ ∈ Lc(f). n (ii)⇒(iii) Let x0 ∈ R be arbitrary and let {xk} be a sequence such that xk → x0. To arrive at a contradiction, assume that f is not l.s.c. at x0, i.e., lim infk→∞ f(xk) < f(x0) Then, there exist a scalar γ and a subsequence {xk}K ⊂ {xk} such that f(xk) ≤ γ < f(x0) for all k ∈ K yielding that {xk}K ⊂ Lγ(f). Since xk → x0 and the set Lγ(f) is closed, it follows that x0 ∈ Lγ(f). Hence, f(x0) ≤ γ - a contradiction. Thus, we must have f(x0) ≤ lim infk→∞ f(xk) Convex Optimization 10 Lecture 4 Proof of Theorem - continues (iii)⇒(i) To arrive at a contradiction assume that epif is not closed. Then, there exists a sequence {(xk, wk)} ⊂ epif such that (xk, wk) → (¯x, w¯) and (¯x, w¯) 6∈ epif . Since (xk, wk) ∈ epif for all k, we have f(xk) ≤ wk for all k Taking the limit inferior as k → ∞, and using wk → w¯, we obtain lim inf f(xk) ≤ lim wk =w. ¯ k→∞ k→∞ Since (¯x, w¯) 6∈ epif, we have f(¯x) > w¯, implying that lim inf f(xk) ≤ w¯ < f(¯x) k→∞ On the other hand, because xk → x¯, and f is l.s.c. at x¯, we have f(¯x) ≤ lim inf f(xk) k→∞ - a contradiction. Hence, epif must be closed. Convex Optimization 11 Lecture 4 Operations Preserving Closedness • Positive Scaling For a closed function f : Rn → R∪{−∞, +∞} and λ > 0, the function g(x) = λf(x) is closed • Sum n For closed functions fi : R → R ∪ {−∞, +∞}, i = 1, . , m, the sum Pm g(x) = i=1 fi(x) is closed • Composition with Affine Mapping For an m × n matrix A, a vector b ∈ Rm, and a closed function f : Rm → R ∪ {−∞, +∞}, the function g(x) = f(Ax + b) is closed This generalizes to a Composition with Continuous Mapping Convex Optimization 12 Lecture 4 Pointwise Supremum n For a collection of closed functions fi : R → R ∪ {−∞, +∞} over an arbitrary index set I, the function g(x) = sup fi(x) is closed i∈I • Example: Piecewise-linear function (polyhedral function) 0 0 f(x) = max{a1x + b1, . , amx + bm} is closed n where ai ∈ R and bi ∈ R for all i Convex Optimization 13 Lecture 4 Convexity and Continuity • Let f : Rn → R with domf = Rn. If f is convex, then f is continuous over Rn • In this case, a convex f is l.s.c Theorem Let f : Rn → R be convex and such that int(domf) 6= ∅. Then, f is continuous over int(domf). • In this case, a convex f need not be l.s.c over Rn • Example: 1 for x = 0 f(x) = 0 for x > 0 +∞ otherwise Convex Optimization 14 Lecture 4 Proof of the Theorem • Using the translation if necessary, we may assume without loss of generality that the origin is in the interior of the domain of f. • It is sufficient to show that f is continuous at the origin • By scaling the unit box if necessary, we may assume without loss of n generality that the unit box {x ∈ R | kxk∞ ≤ 1} is contained in domf n • Let vi, i ∈ I = {1,..., 2 } be vertices of the unit box (i.e., each vi has entries 1 or −1). The unit box can be viewed as a simplex generated by these vertices, i.e., every x with kxk∞ ≤ 1 is a convex combination of vertices vi, i ∈ I Convex Optimization 15 Lecture 4 or equivalently: every x with kxk∞ ≤ 1 is given by X P x = αivi with αi ≥ 0 and i∈I αi = 1 i∈I • Note that by convexity of f, we have f(x) ≤ max f(vi) = M (1) i∈I • Let xk → 0 and assume that xk 6= 0 for all k xk −xk • We introduce yk = and zk = kxkk∞ kxkk∞ • Note that we can write 0 as a convex combination of yk and zk, as follows 1 kxkk∞ 0 = xk + zk for all k kxkk∞ + 1 kxkk∞ + 1 Convex Optimization 16 Lecture 4 • By convexity of f it follows that 1 kxkk∞ f(0) ≤ f(xk) + f(zk) for all k kxkk∞ + 1 kxkk∞ + 1 • By letting k → ∞ and using Eq. (1), we have f(0) ≤ lim inf f(xk) (2) k→0 • Note that we can write xk = (1 − kxkk∞) 0 + kxkk∞ yk for all k • By using convexity, we obtain f(xk) ≤ (1 − kxkk∞) f(0) + kxkk∞ f(yk) Convex Optimization 17 Lecture 4 • Taking the limsup as k → ∞ and using Eq.