Lecture 4

Closed Functions

September 4, 2007 Lecture 4 Outline

• Log-Transformation

• Jensen’s Inequality

• Level Sets

• Closed Functions

• Convexity and Continuity

Convex Optimization 1 Lecture 4 Log-Transformation of Variables

Useful for transforming a nonconvex function to a convex one

A posynomial is a function of y1 > 0, . . . , yn > 0 of the form

a1 an g(y1, . . . , yn) = by1 ··· yn

with scalars b > 0 and ai > 0 for all i. • A posynomial need not be convex • Log-transformation of variables

xi = ln yi for all i • We have a convex function

0 f(x) = bea1x1 ··· eanxn = bea x

Convex Optimization 2 Lecture 4 Log-Transformation of Functions

Replacing f with ln f [when f(x) > 0 over domf] Useful for:

• Transforming non-separable functions to separable ones

n 1/n Example: (Geometric Mean) f(x) = (Πi=1xi) for x with xi > 0 for all i is non-separable. Using F (x) = ln f(x), we obtain a separable F ,

n 1 X F (x) = ln xi n i=1

• Separable structure of objective function is advantageous in distributed optimization

Convex Optimization 3 Lecture 4 General Convex Inequality Basic convex inequality: For a convex f, we have for x, y ∈ domf and α ∈ (0, 1),

f(αx + (1 − α)y) ≤ αf(x) + (1 − α)f(y)

General convex inequality: For a convex f and any convex combination of the points in domf, we have

m ! m X X f αixi ≤ αif(xi) i=1 i=1

Pm (xi ∈ domf and αi > 0 for all i, i=1 αi = 1, m > 0 integer) Pm A convex combination i=1 αixi can be viewed as the expectation of a random vector z having outcomes z = xi with probability αi

Convex Optimization 4 Lecture 4 Jensen’s Inequality

General convex inequality can be interpreted as:

• For a convex f and a (finite) discrete random variable z with outcomes

zi ∈ domf, we have f(Ez) ≤ E [f(z)]

General Jensen’s inequality: The above relation holds for a convex f and, a random variable z with outcomes in domf and a finite expectation Ez

Convex Optimization 5 Lecture 4 Level Sets

Def. Given a scalar c ∈ R and a function f : Rn → R, a (lower) level set of f associated with c is given by

Lc(f) = {x ∈ domf | f(x) ≤ c}

2 n x Examples: f(x) = kxk for x ∈ R , f(x1, x2) = e 1 • Every level set of a convex function is convex

• Converse is false: Consider f(x) = −ex for x ∈ R Def. A function g is concave when −g is convex

• Every (upper) level set of a concave function is convex

Convex Optimization 6 Lecture 4 Convex Extended-Value Functions

• The definition of convexity that we have used thus far is applicable to functions mapping from a subset of Rn to Rn. It does not apply to extended-value functions mapping from a subset of Rn to the extended set R ∪ {−∞, +∞}. • The general definition of convexity relies on the epigraph of a function • Let f be a function taking values in R ∪ {−∞, +∞}. The epigraph of f is the set given by

n n epif = {(x, w) ∈ R × R | f(x) ≤ w for some x ∈ R }.

• Def. The extended-value function f is convex when its epigraph epif is convex set (in Rn × R). • This definition coincides with the one we have used for a function f with values in R.

Convex Optimization 7 Lecture 4 Closed Functions

Def. A function f is closed if its epigraph epif is a closed set in Rn × R, i.e., for every sequence {(xk, wk)} ⊂ epif converging to some (ˆx, wˆ), we have (ˆx, wˆ) ∈ epif Examples Affine functions are closed [f(x) = a0x + b] Quadratic functions are closed [f(x) = x0P x + a0x + b] Continuous functions are closed • A class of closed functions is larger than the class of continuous functions

• For example f(x) = 0 for x = 0, f(x) = 1 for x > 0, and f(x) = +∞ otherwise This f is closed but not continuous

Convex Optimization 8 Lecture 4 Closed Function Properties

Lower-Semicontinuity Def. A function f is lower-semicontinuous at

a given vector x0 if for every sequence {xk} converging to x0, we have f(x0) ≤ lim infk→0 f(xk) We say that f is lower-semicontinuous over a set X if f is lower- semicontinuous at every x ∈ X

Th. For a function f : Rn → R ∪ {−∞, +∞} the following statements are equivalent (i) f is closed

(ii) Every level set of f is closed

(iii) f is lower-semicontinuous (l.s.c.) over Rn

Convex Optimization 9 Lecture 4 Proof of Theorem (i)⇒(ii) Let c be any scalar and consider Lc(f). If Lc(f) = ∅, then Lc(f) is closed. Suppose now that Lc(f) 6= ∅. Pick {xk} ⊂ Lc(f) such that n xk → x¯ for some x¯ ∈ R . We have f(xk) ≤ c for all k, implying that (xk, c) ∈ epif for all k. Since (xk, c) → (¯x, c) and epif is closed, it follows that (¯x, c) ∈ epif. Consequently f(¯x) ≤ c, showing that x¯ ∈ Lc(f). n (ii)⇒(iii) Let x0 ∈ R be arbitrary and let {xk} be a sequence such that xk → x0. To arrive at a contradiction, assume that f is not l.s.c. at x0, i.e.,

lim infk→∞ f(xk) < f(x0) Then, there exist a scalar γ and a subsequence {xk}K ⊂ {xk} such that f(xk) ≤ γ < f(x0) for all k ∈ K yielding that {xk}K ⊂ Lγ(f). Since xk → x0 and the set Lγ(f) is closed, it follows that x0 ∈ Lγ(f). Hence, f(x0) ≤ γ - a contradiction. Thus, we must have

f(x0) ≤ lim infk→∞ f(xk)

Convex Optimization 10 Lecture 4 Proof of Theorem - continues

(iii)⇒(i) To arrive at a contradiction assume that epif is not closed.

Then, there exists a sequence {(xk, wk)} ⊂ epif such that (xk, wk) → (¯x, w¯) and (¯x, w¯) 6∈ epif . Since (xk, wk) ∈ epif for all k, we have f(xk) ≤ wk for all k Taking the limit inferior as k → ∞, and using wk → w¯, we obtain

lim inf f(xk) ≤ lim wk =w. ¯ k→∞ k→∞ Since (¯x, w¯) 6∈ epif, we have f(¯x) > w¯, implying that

lim inf f(xk) ≤ w¯ < f(¯x) k→∞ On the other hand, because xk → x¯, and f is l.s.c. at x¯, we have

f(¯x) ≤ lim inf f(xk) k→∞ - a contradiction. Hence, epif must be closed.

Convex Optimization 11 Lecture 4 Operations Preserving Closedness

• Positive Scaling For a closed function f : Rn → R∪{−∞, +∞} and λ > 0, the function g(x) = λf(x) is closed

• Sum n For closed functions fi : R → R ∪ {−∞, +∞}, i = 1, . . . , m, the sum Pm g(x) = i=1 fi(x) is closed

• Composition with Affine Mapping For an m × n matrix A, a vector b ∈ Rm, and a closed function f : Rm → R ∪ {−∞, +∞}, the function g(x) = f(Ax + b) is closed

This generalizes to a Composition with Continuous Mapping

Convex Optimization 12 Lecture 4 Pointwise Supremum

n For a collection of closed functions fi : R → R ∪ {−∞, +∞} over an arbitrary index set I, the function

g(x) = sup fi(x) is closed i∈I • Example: Piecewise-linear function (polyhedral function)

0 0 f(x) = max{a1x + b1, . . . , amx + bm} is closed

n where ai ∈ R and bi ∈ R for all i

Convex Optimization 13 Lecture 4 Convexity and Continuity

• Let f : Rn → R with domf = Rn.

If f is convex, then f is continuous over Rn • In this case, a convex f is l.s.c

Theorem Let f : Rn → R be convex and such that int(domf) 6= ∅. Then, f is continuous over int(domf).

• In this case, a convex f need not be l.s.c over Rn

• Example:   1 for x = 0  f(x) = 0 for x > 0   +∞ otherwise

Convex Optimization 14 Lecture 4 Proof of the Theorem

• Using the translation if necessary, we may assume without loss of generality that the origin is in the interior of the domain of f.

• It is sufficient to show that f is continuous at the origin

• By scaling the unit box if necessary, we may assume without loss of n generality that the unit box {x ∈ R | kxk∞ ≤ 1} is contained in domf

n • Let vi, i ∈ I = {1,..., 2 } be vertices of the unit box (i.e., each vi has entries 1 or −1). The unit box can be viewed as a simplex generated by these vertices, i.e.,

every x with kxk∞ ≤ 1 is a convex combination of vertices vi, i ∈ I

Convex Optimization 15 Lecture 4

or equivalently: every x with kxk∞ ≤ 1 is given by

X P x = αivi with αi ≥ 0 and i∈I αi = 1 i∈I

• Note that by convexity of f, we have

f(x) ≤ max f(vi) = M (1) i∈I

• Let xk → 0 and assume that xk 6= 0 for all k xk −xk • We introduce yk = and zk = kxkk∞ kxkk∞ • Note that we can write 0 as a convex combination of yk and zk, as follows

1 kxkk∞ 0 = xk + zk for all k kxkk∞ + 1 kxkk∞ + 1

Convex Optimization 16 Lecture 4

• By convexity of f it follows that

1 kxkk∞ f(0) ≤ f(xk) + f(zk) for all k kxkk∞ + 1 kxkk∞ + 1

• By letting k → ∞ and using Eq. (1), we have

f(0) ≤ lim inf f(xk) (2) k→0

• Note that we can write xk = (1 − kxkk∞) 0 + kxkk∞ yk for all k

• By using convexity, we obtain

f(xk) ≤ (1 − kxkk∞) f(0) + kxkk∞ f(yk)

Convex Optimization 17 Lecture 4

• Taking the limsup as k → ∞ and using Eq. (1), we see that

lim sup f(xk) ≤ f(0) k→∞

• From this relation and Eq. (2), we see that

lim f(xk) = f(0), k→∞

showing that f is continuous at 0

Convex Optimization 18