Convex Functions

Prof. Daniel P. Palomar

ELEC5470/IEDA6100A - Convex Optimization The Hong Kong University of Science and Technology (HKUST) Fall 2020-21 Outline of Lecture

• Deﬁnition convex function

• Examples on R, Rn, and Rn×n

• Restriction of a convex function to a line

• First- and second-order conditions

• Operations that preserve convexity

• Quasi-convexity, log-convexity, and convexity w.r.t. generalized inequalities

(Acknowledgement to Stephen Boyd for material for this lecture.)

Daniel P. Palomar 1 Deﬁnition of Convex Function

• A function f : Rn −→ R is said to be convex if the domain, dom f, is convex and for any x, y ∈ dom f and 0 ≤ θ ≤ 1,

f (θx + (1 − θ) y) ≤ θf (x) + (1 − θ) f (y) .

• f is strictly convex if the inequality is strict for 0 < θ < 1.

• f is concave if −f is convex.

Daniel P. Palomar 2 Examples on R Convex functions: • aﬃne: ax + b on R • powers of absolute value: |x|p on R, for p ≥ 1 (e.g., |x|)

p 2 • powers: x on R++, for p ≥ 1 or p ≤ 0 (e.g., x ) • exponential: eax on R

• negative entropy: x log x on R++ Concave functions: • aﬃne: ax + b on R

p • powers: x on R++, for 0 ≤ p ≤ 1

• logarithm: log x on R++

Daniel P. Palomar 3 Examples on Rn

• Aﬃne functions f (x) = aT x + b are convex and concave on Rn.

n • Norms kxk are convex on R (e.g., kxk∞, kxk1, kxk2).

• Quadratic functions f (x) = xT P x + 2qT x + r are convex Rn if and only if P 0.

Qn 1/n n • The geometric mean f (x) = ( i=1 xi) is concave on R++.

P xi n • The log-sum-exp f (x) = log i e is convex on R (it can be used to approximate max xi). i=1,...,n

2 n • Quadratic over linear: f (x, y) = x /y is convex on R × R++.

Daniel P. Palomar 4 Examples on Rn×n • Aﬃne functions: (prove it!) f (X) = Tr (AX) + b

are convex and concave on Rn×n. • Logarithmic determinant function: (prove it!) f (X) = logdet (X)

is concave on Sn = {X ∈ Rn×n | X 0}. • Maximum eigenvalue function: (prove it!) yT Xy f (X) = λmax (X) = sup T y6=0 y y is convex on Sn.

Daniel P. Palomar 5 Epigraph • The epigraph of f if the set

epi f = (x, t) ∈ Rn+1 | x ∈ dom f, f (x) ≤ t .

• Relation between convexity in sets and convexity in functions:

f is convex ⇐⇒ epi f is convex

Daniel P. Palomar 6 Restriction of a Convex Function to a Line

• f : Rn −→ R is convex if and only if the function g : R −→ R

g (t) = f (x + tv) , dom g = {t | x + tv ∈ dom f}

is convex for any x ∈ dom f, v ∈ Rn.

• In words: a function is convex if and only if it is convex when restricted to an arbitrary line.

• Implication: we can check convexity of f by checking convexity of functions of one variable!

• Example: concavity of logdet (X) follows from concavity of log (x).

Daniel P. Palomar 7 Example: concavity of logdet (X) : g (t) = logdet (X + tV ) = logdet (X) + logdet I + tX−1/2VX−1/2

n X = logdet (X) + log (1 + tλi) i=1

−1/2 −1/2 where λi’s are the eigenvalues of X VX . The function g is concave in t for any choice of X 0 and V ; therefore, f is concave.

Daniel P. Palomar 8 First and Second Order Condition

• Gradient (for diﬀerentiable f):

h iT ∇f (x) = ∂f(x) ··· ∂f(x) ∈ Rn. ∂x1 ∂xn

• Hessian (for twice diﬀerentiable f):

∂2f (x) ∇2f (x) = ∈ Rn×n. ∂xi∂xj ij

• Taylor series: 1 f (x + δ) = f (x) + ∇f (x)T δ + δT ∇2f (x) δ + o kδk2 . 2

Daniel P. Palomar 9 • First-order condition: a diﬀerentiable f with convex domain is convex if and only if

f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ dom f

• Interpretation: ﬁrst-order approximation if a global underestimator.

• Second-order condition: a twice diﬀerentiable f with convex domain is convex if and only if

∇2f (x) 0 ∀x ∈ dom f

Daniel P. Palomar 10 Examples

• Quadratic function: f (x) = (1/2) xT P x + qT x + r (with P ∈ Sn)

∇f (x) = P x + q, ∇2f (x) = P

is convex if P 0.

2 • Least-squares objective: f (x) = kAx − bk2

∇f (x) = 2AT (Ax − b) , ∇2f (x) = 2AT A

is convex.

Daniel P. Palomar 11 • Quadratic-over-linear: f (x, y) = x2/y

2 y ∇2f (x, y) = y −x 0 y3 −x

is convex for y > 0.

Daniel P. Palomar 12 Operations that Preserve Convexity How do we establish the convexity of a given function?

1. Applying the deﬁnition.

2. With ﬁrst- or second-order conditions.

3. By restricting to a line.

4. Showing that the functions can be obtained from simple functions by operations that preserve convexity: • nonnegative weighted sum • composition with aﬃne function (and other compositions) • pointwise maximum and supremum, minimization • perspective

Daniel P. Palomar 13 • Nonnegative weighted sum: if f1, f2 are convex, then α1f1 +α2f2 is convex, with α1, α2 ≥ 0.

• Composition with aﬃne functions: if f is convex, then f (Ax + b) is convex (e.g., ky − Axk is convex, logdet I + HXHT is concave).

• Pointwise maximum: if f1, . . . , fm are convex, then f (x) = max {f1, . . . , fm} is convex.

n Example: sum of r largest components of x ∈ R : f (x) = x[1] + x[2] + ··· + x[r] where x[i] is the ith largest component of x.

Proof: f (x) = max {xi1 + xi2 + ··· + xir | 1 ≤ i1 < i2 < ··· < ir ≤ n}.

Daniel P. Palomar 14 • Pointwise supremum: if f (x, y) is convex in x for each y ∈ A, then g (x) = sup f (x, y) y∈A is convex.

Example: distance to farthest point in a set C:

f (x) = sup kx − yk . y∈C

Example: maximum eigenvalue of symmetric matrix: for X ∈ Sn,

yT Xy λmax (X) = sup T . y6=0 y y

Daniel P. Palomar 15 • Composition with scalar functions: let g : Rn −→ R and h : R −→ R, then the function f (x) = h (g (x)) satisﬁes: g convex, h convex nondecreasing f (x) is convex if g concave, h convex nonincreasing

• Minimization: if f (x, y) is convex in (x, y) and C is a convex set, then g (x) = inf f (x, y) y∈C is convex (e.g., distance to a convex set).

Example: distance to a set C:

f (x) = inf kx − yk y∈C is convex if C is convex.

Daniel P. Palomar 16 • Perspective: if f (x) is convex, then its perspective

g (x, t) = tf (x/t) , dom g = (x, t) ∈ Rn+1 | x/t ∈ dom f, t > 0

is convex.

Example: f (x) = xT x is convex; hence g (x, t) = xT x/t is convex for t > 0.

Example: the negative logarithm f (x) = − log x is convex; hence the 2 relative entropy function g (x, t) = t log t − t log x is convex on R++.

Daniel P. Palomar 17 Quasi-Convexity Functions

• A function f : Rn −→ R is quasi-convex if dom f is convex and the sublevel sets Sα = {x ∈ dom f | f (x) ≤ α} are convex for all α.

• f is quasiconcave if −f is quasiconvex.

Daniel P. Palomar 18 Examples:

• p|x| is quasiconvex on R

• ceil (x) = inf {z ∈ Z | z ≥ x} is quasilinear

• log x is quasilinear on R++

2 • f (x1, x2) = x1x2 is quasiconcave on R++

• the linear-fractional function

aT x + b f (x) = , dom f = x | cT x + d > 0 cT x + d is quasilinear

Daniel P. Palomar 19 Log-Convexity

• A positive function f is log-concave is log f is concave:

f (θx + (1 − θ) y) ≥ f (x)θ f (y)1−θ for 0 ≤ θ ≤ 1.

• f is log-convex if log f is convex.

a • Example: x on R++ is log-convex for a ≤ 0 and log-concave for a ≥ 0

• Example: many common probability densities are log-concave

Daniel P. Palomar 20 Convexity w.r.t. Generalized Inequalities

• f : Rn −→ Rm is K-convex if dom f is convex and for any x, y ∈ dom f and 0 ≤ θ ≤ 1,

f (θx + (1 − θ) y) K θf (x) + (1 − θ) f (y) .

m m 2 m • Example: f : S −→ S , f (X) = X is S+ -convex.

Daniel P. Palomar 21 References

Chapter 3 of

• Stephen Boyd and Lieven Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge University Press, 2004.

https://web.stanford.edu/˜boyd/cvxbook/bv cvxbook.pdf

Daniel P. Palomar 22