Convex Optimization 3

Home , Convex function, Hypograph (mathematics), Quasiconvex function

Convex Optimization 3. Convex Functions

Prof. Ying Cui

Department of Electrical Engineering Shanghai Jiao Tong University

2018

SJTU Ying Cui 1 / 42 Outline

Basic properties and examples

Operations that preserve convexity

The conjugate function

Quasiconvex functions

Log-concave and log-convex functions

Convexity with respect to generalized inequalities

SJTU Ying Cui 2 / 42 Deﬁnition

◮ convex: f : Rn R is convex if domf is a convex set and if → f (θx + (1 θ)y) θf (x) + (1 θ)f (y) − ≤ − for all x, y domf , and θ with 0 θ 1 ∈ ≤ ≤ ◮ geometric interpretation: line segment between (x, f (x)) and (y, f (y)) (i.e., chord from x to y) lies above graph of f

(y, f(y))

(x, f(x))

Figure 3.1 Graph of a convex function. The chord (i.e. , line segment) between any two points on the graph lies above the graph.

◮ concave: f is concave if f is convex −

SJTU Ying Cui 3 / 42 Deﬁnition

◮ strictly convex: f : Rn R is strictly convex if domf is a → convex set and if

f (θx + (1 θ)y) <θf (x) + (1 θ)f (y) − − for all x, y domf , x = y, and θ with 0 <θ< 1 ∈ 6 ◮ strictly concave: f is strictly concave if f is strictly convex − ◮ aﬃne functions are both convex and concave ◮ any function that is convex and concave is aﬃne

SJTU Ying Cui 4 / 42 Examples on R

convex: ◮ aﬃne: ax+b on R, for any a, b R ∈ ◮ exponential: eax on R, for any a R ∈ ◮ powers: xα on R , for α 1 or α 0 ++ ≥ ≤ ◮ powers of absolute value: x p on R, for p 1 ◮ | | ≥ negative entropy: x log x on R++ concave: ◮ aﬃne: ax+b on R, for any a, b R ∈ ◮ α powers: x on R++, for 0 α 1 ◮ ≤ ≤ logarithm: log x on R++

SJTU Ying Cui 5 / 42 n m n Examples on R and R × Examples on Rn: ◮ aﬃne function f (x)=aT x+b is both convex and concave ◮ every norm is convex ◮ due to triangle inequality and homogeneity ◮ n p 1/p n lp-norms: x p=( xi ) for p 1 ( x = xi , || || i=1 | | ≥ || ||1 i=1 | | x ∞=maxk xk ) ◮ || || | | P P max function f (x) = max x , , xn is convex { 1 · · · } ◮ log-sum-exp f (x) = log(ex1 + + exn ) is convex · · · ◮ a diﬀerentiable approximation of the max function:

x1 xn x1 xn log(e + +e ) log n max x , , xn log(e + +e ) ··· − ≤ { 1 ··· }≤ ··· m n Examples on R × : ◮ T m n aﬃne function f (X )= tr(A X )+ b = i=1 j=1 Aij Xij + b is both convex and concave P P ◮ spectral (maximum singular value) norm f (X )= X = σ (X ) = (λ (X T X ))1/2 on is convex k k2 max max SJTU Ying Cui 6 / 42 Restriction of a convex function to a line ◮ a function f : Rn R is convex iﬀ it is convex when restricted to any line→ that intersects its domain, i.e., ◮ g(t)= f (x + tv) is convex on t x + tv domf for all x domf and all v Rn { | ∈ } ∈ ∈ ◮ check convexity of a function of multiple variables by restricting it to a line and checking convexity of a function of one variable ◮ example: f : Sn R with f (X ) = logdet X , domf = Sn → ++ Consider an arbitrary line X = Z + tV Sn with Z, V Sn. ∈ ++ ∈ w. l. o. g., assume t = 0 is in the interval, i.e., Z Sn . ∈ ++ 1/2 1/2 1/2 1/2 g(t) = log det(Z + tV ) = log det(Z (I + tZ − VZ − )Z ) 1/2 1/2 =logdet Z + log det(I + tZ − VZ − ) n 1/2 1/2 =logdet Z + log(1 + tλi ) λi : eigenvalues of Z − VZ − i X=1 g is concave in t. Thus, f is concave.

SJTU Ying Cui 7 / 42 Extended-value extension ◮ extended-value extension f˜ of a convex function f is

f (x), x domf f˜(x)= ∈ , x / domf (∞ ∈

◮ f˜ is defined on all Rn, and takes values in R ◮ recover domain of f from f˜ as domf = x f˜∪(x {∞}) < { | ∞} ◮ extension can simplify notation, as no need to explicitly describe the domain, or add the qualifier ‘for all x domf ’ ∈ ◮ basic defining inequality for convexity can be expressed as: for 0 <θ< 1, f˜(θx + (1 θ)y) θf˜(x) + (1 θ)f˜(y) for any x − ≤ − and y ◮ the inequality always holds for θ =0, 1 ◮ no need to mention the two conditions: domf is convex (can be shown by contradiction) and x, y domf (x, y Rn is used instead, which can be omitted) ∈ ∈

SJTU Ying Cui 8 / 42 First-order conditions Suppose f is differentiable, i.e., domf is open and gradient f (x)= ∂f (x) , , ∂f (x) exists at any x domf ∇ ∂x1 · · · ∂xn ∈ ◮ f is convex iff domf is convex and f (y) f (x)+ f (x)T (y x) for all x, y domf ≥ ∇ − ∈ ◮ first-order Taylor approx. of a convex function is a global underestimator of it; if first-order Taylor approx. of a function is always a global underestimator of it, then it is convex ◮ local information about a convex function (value and derivative at a point) implies global information (a global underestimator) ◮ if f is convex and f (x) = 0, then x is a global minimizer of f ∇ ◮ f is strictly convex iff domf is convex and f (y) > f (x)+ f (x)T (y x) for all x, y domf and x = y ∇ − ∈ 6 f(y)

f(x) + ∇f(x)T (y − x)

(x, f(x))

Figure 3.2 If f is convex and diﬀerentiable, then f(x)+∇f(x)T (y−x) ≤ f(y) for all x, y ∈ dom f. SJTU Ying Cui 9 / 42 Second-order conditions

Suppose f is twice diﬀerentiable, i.e., domf is open and Hessian 2 n 2 ∂2f (x) f (x) S exists at any x domf , where f (x)ij = , ∇ ∈ ∈ ∇ ∂xi ∂xj i, j = 1, , n · · · ◮ f is convex iﬀ domf is convex and 2f (x) 0 for all ∇ x domf ∈ ◮ for a function on R, this reduces to domf is an interval and f ′′(x) 0 for all x in the interval ◮ 2f (x≥) 0 means the graph of f has positive (upward) ∇curvature at x ◮ if domf is convex and 2f (x) 0 for all x domf , then f is strictly convex ∇ ≻ ∈ ◮ the converse is not true, e.g., f (x)= x 4 is strictly convex but f ′′(0) = 0

SJTU Ying Cui 10 / 42 Second-order conditions Examples ◮ quadratic function: f (x) = (1/2)xT Px + qT x + r (P Sn) ∈ f (x)= Px + q, 2f (x)= P ∇ ∇ convex iﬀ P Sn ∈ + ◮ least-squares objective: f (x)= Ax b 2 = xT AT Ax 2xT AT b + bT b k − k2 − f (x) = 2AT (Ax b), 2f (x) = 2AT A ∇ − ∇ convex for all A Rm n (as AT A 0 for all A Rm n) ∈ × ∈ × ◮ quadratic-over-linear function: f (x, y)= x2/y

2 y y T 2f (x, y)= 0 ∇ y 3 x x − − convex for all x R and y R (as zzT 0 for all z Rn) ∈ ∈ ++ ∈ SJTU Ying Cui 11 / 42 Second-order conditions Examples ◮ n log-sum-exp: f (x) =log k=1 exp xk is convex ◮ proof: P

2 1 1 T f (x)= diag(z) zz (zk = exp xk ) ∇ 1T z − (1T z)2

to show 2f (x) 0, we must verify that v T 2f (x)v 0 for all v: ∇ ∇ ≥

2 2 ( zk v )( zk ) ( vk zk ) v T 2f (x)v = k k k − k 0 ∇ ( z )2 ≥ P P k k P 2 2 P since ( k vk zk ) ( k zk vk )( k zk ) (from Cauchy-Schwarz T ≤T T 2 inequality (a a)(b b) (a b) by treating ai = vi √zi and P P≥ P bi = √zi ) ◮ n 1/n n geometric mean: f (x) = ( k=1 xk ) on R++ is concave (similar proof as for log-sum-exp) Q SJTU Ying Cui 12 / 42 Sublevel set and superlevel set

Sublevel set ◮ α-sublevel set of f : Rn R: x domf f (x) α → { ∈ | ≤ } ◮ sublevel sets of a convex function are convex ◮ the converse is false (e.g., f (x)= exp x is not convex (indeed, strictly concave) but all its− sublevel sets are convex) Superlevel set ◮ α-superlevel set of f : Rn R: x domf f (x) α → { ∈ | ≥ } ◮ superlevel sets of a concave function are convex To establish convexity of a set, express it as a sublevel set of a convex function, or as the superlevel set of a concave function.

SJTU Ying Cui 13 / 42 Epigraph and hypergraph ◮ graph of f : Rn R: → (x, f (x)) x domf Rn+1 { | ∈ }⊆ ◮ epigraph of f : Rn R: → epi f = (x, t) Rn+1 x domf , f (x) t Rn+1 { ∈ | ∈ ≤ }⊆ ◮ f is convex iﬀ epi f is a convex set ◮ hypograph of f : Rn R: → hypo f = (x, t) Rn+1 x domf , f (x) t Rn+1 { ∈ | ∈ ≥ }⊆ ◮ f is concave iﬀ hypo f is a convex set

epi f

Figure 3.5 Epigraph of a function f, shown shaded. The lower boundary, shown darker, is the graph of f.

SJTU Ying Cui 14 / 42 Jensen’s inequality and extensions

◮ basic inequality: if f is convex, x, y domf and 0 θ 1, ∈ ≤ ≤ then f (θx + (1 θ)y) θf (x) + (1 θ)f (y) − ≤ − ◮ extension to convex combinations of more than two points: if f is convex, x , , xk domf , and θ , ,θk 0 with 1 · · · ∈ 1 · · · ≥ θ + + θk = 1, then 1 · · · f (θ x + + θk xk ) θ f (x )+ + θk f (xk ) 1 1 · · · ≤ 1 1 · · · ◮ extensions to inﬁnite sums and integrals (if p(x) 0 on ≥ S domf , p(x)dx = 1, then ⊆ S f p(x)xdx f (x)p(x)dx, provided the integrals exist) S R S ◮ ≤ extensionR to expected R values: if f is convex and X is a random variable such that X domf w.p. 1, then ∈ f (EX ) Ef (X ), provided the expectations exist ≤ ◮ many famous inequalities (e.g., arithmetic-geometric mean inequality and H¨older’s inequality) can be derived by applying Jensen’s inequality to some convex function

SJTU Ying Cui 15 / 42 Operations that preserve convexity

practical methods for establishing convexity of a function ◮ verify definition (often simplified by restricting to a line) ◮ for twice differentiable functions, show 2f (x) 0 ∇ ◮ show that f is obtained from simple convex functions by operations that preserve convexity ◮ nonnegative weighted sum ◮ composition with affine function ◮ pointwise maximum and supremum ◮ composition ◮ minimization ◮ perspective

SJTU Ying Cui 16 / 42 Nonnegative weighted sums

◮ nonnegative multiple: if f is convex and α 0, then αf is ≥ convex ◮ sum: if f1 and f2 are convex, then f1 + f2 is convex ◮ nonnegative weighted sum: if f , , fm are convex and 1 · · · w , , wm 0, then w f + + wmfm is convex 1 · · · ≥ 1 1 · · · ◮ extension to inﬁnite sums ◮ extension to integrals: if f (x, y) is convex in x for each y , ∈ A and w(y) 0 for each y , then g(x)= A w(y)f (x, y)dy is convex in≥ x, provided the∈ A integral exists R

SJTU Ying Cui 17 / 42 Composition with an aﬃne function

Suppose f : Rn R, A Rn m, and b Rn. Deﬁne g : Rm R → ∈ × ∈ → by g(x)= f (Ax + b), domg = x Ax + b domf { | ∈ } Then if f is convex, so is g; if f is concave, so is g. examples ◮ log barrier for linear inequalities

m T T f (x)= log(bi a x), domf = x a x < bi , i = 1, ..., m − − i { | i } i X=1 is convex ◮ (any) norm of aﬃne function f (x)= Ax + b is convex || ||

SJTU Ying Cui 18 / 42 Pointwise maximum

If f1, ..., fm are convex, then their pointwise maximum f , deﬁned by

f (x) = max f (x), ...., fm(x) , domf = domf domfm { 1 } 1 ∩···∩ is convex. (over a ﬁnite set of convex functions) examples ◮ T piecewise-linear function: f (x) = maxi=1,...,m(ai x + bi ) is convex ◮ sum of r largest components of x Rn: f (x)= r x is ∈ i=1 [i] convex, where x[i] is ith largest component of x ◮ P proof: f (x) = max xi1 + + xir 1 i1 < i2 < < ir n { ··· n | ≤ ··· ≤ } is the pointwise maximum of r linear functions ◮ extension: if w1 w2 wr 0, then r ≥ ≥···≥ ≥ f (x)= i=1 wi x[i] is convex ◮ proof: P r r−1 f (x)= wr i=1 x[i] +(wr−1 wr ) i=1 x[i] + +(w2 w1)x[1] is a nonnegtive weighted sum− of (r + 1) convex··· functions− P P SJTU Ying Cui 19 / 42 Pointwise supremum If f (x, y) is convex in x for each y A, then ∈ g(x) = sup f (x, y), domg = x (x, y) domf y A, g(x) < y A { | ∈ ∀ ∈ ∞} ∈ is convex in x. (over an inﬁnite set of convex functions) examples ◮ n T support function of set C R , C = : SC (x) = sup y x is ⊆ 6 ∅ y C convex ∈ ◮ distance to farthest point in set C Rn: f (x) = sup x y ⊆ y C || − || is convex ∈ ◮ maximum eigenvalue of symmetric matrix: T n λmax (X )= sup y Xy with domf = S is convex y =1 || ||2 ◮ spectral norm (maximum singular value) of a matrix: T p q X 2 = sup u Xv with domf = R × is convex || || u =1, v =1 || ||2 || ||2 SJTU Ying Cui 20 / 42 Scalar Composition

composition of g : Rn R and h : R R: → → f (x)= h(g(x)), domf = x domg g(x) domh { ∈ | ∈ } h convex, h˜ nondecreasing, g convex f is convex if (h convex, h˜ nonincreasing, g concave h concave, h˜ nondecreasing, g concave f is concave if (h concave, h˜ nonincreasing, g convex ◮ proof (n = 1, g, h twice diﬀerentiable, domg = domh = R)

2 f ′′(x)= h′′(g(x))g ′(x) + h′(g(x))g ′′(x)

◮ note: monotonicity must hold for extended-value extension h˜

SJTU Ying Cui 21 / 42 Scalar Composition

examples ◮ if g is convex then exp g(x) is convex ◮ if g is concave and positive, then log g(x) is concave ◮ if g is concave and positive, then 1/g(x) is convex ◮ if g is convex and nonnegative and p 1, then g(x)p is ≥ convex ◮ if g is convex then log( g(x)) is convex on x g(x) < 0 − − { | }

SJTU Ying Cui 22 / 42 Vector composition

composition of g : Rn Rk and h : Rk R: → →

f (x)= h(g(x)) = h(g1(x), g2(x), ..., gk (x))

f is convex if h convex, h˜ nondecreasing in each argument, gi convex ˜ (h convex, h nonincreasing in each argument, gi concave f is concave if h concave, h˜ nondecreasing in each argument, gi concave ˜ (h concave, h nonincreasing in each argument, gi convex ◮ proof (n = 1, g, h twice diﬀerentiable, domg = R, domh = Rk ):

T 2 T f ′′(x)= g ′(x) h(g(x))g ′(x)+ h(g(x)) g ′′(x) ∇ ∇

SJTU Ying Cui 23 / 42 Vector composition

examples ◮ m log( i=1 exp gi (x)) is convex if gi are convex ◮ m g x g i=1Plog i ( ) is concave if i are concave and positive ◮ k p 1/p ( gi (x) ) , 0 < p 1 is concave if gi are concave and P i=1 ≤ nonnegative P ◮ k p 1/p ( gi (x) ) , p 1 is convex if gi are convex and i=1 ≥ nonnegative P ◮ k 1/k ( i=1 gi (x)) is concave if gi are concave and nonnegative Q

SJTU Ying Cui 24 / 42 Minimization If f (x, y) is convex in (x, y) and C is a convex nonempty set, then g(x)= inf f (x, y), domg = x (x, y) domf for some y C y C { | ∈ ∈ } ∈ is convex examples ◮ Schur complement: f (x, y)= xT Ax + 2xT By + y T Cy with AB X = 0, C 0 BT C ≻ T 1 T Thus, g(x) = inf f (x, y)= x (A BC − B )x is convex, y − implying Schur complement of C in matrix X : 1 T A BC − B 0 T − x x ◮ note: f (x, y)= X is convex in (x, y) as X 0 y y ◮ distance to a set: dist (x, S)= inf x y is convex if S is y S || − || convex ∈ ◮ note: x y is convex in in (x, y) || − || SJTU Ying Cui 25 / 42 Perspective of a function perspective of a function f : Rn R is the function → g : Rn R R × → g(x, t)= tf (x/t), domg = (x, t) x/t domf , t > 0 { | ∈ } If f is convex, g is convex; if f is concave, g is concave examples ◮ Euclidean norm squared: perspective of convex function f (x)= xT x on Rn: g(x, t)= t(x/t)T (x/t)= xT x/t is convex in (x, t) for t > 0 ◮ negative logarithm: perspective of convex function f (x)= log x on R : − ++ g(x, t)= t log(x/t)= t log(t/x)= t log t t log x (called − 2 − relative entropy) is convex on R++ ◮ if f is convex, then g(x) = (cT x + d)f ((Ax + b)/(cT x + d)) is convex on domg = x cT x + d > 0, (Ax + b)/(cT x + d) domf { | ∈ } SJTU Ying Cui 26 / 42 Conjugate function conjugate f : Rn R of a function f : Rn R is ∗ → → T f ∗(y)= sup (y x f (x)) x domf − ∈ domf consists of y Rn for which the supremum is ﬁnite, i.e., ∗ ∈ for which the diﬀerence y T x f (x) is bounded above on domf − f(x) xy

(0 , −f ∗(y))

Figure 3.8 A function f : R → R, and a value y ∈ R. The conjugate ∗ function f (y) is the maximum gap between the linear function yx and f(x), as shown by the dashed line in the figure. If f is differentiable, this ′ occurs at a point x where f (x) = y. ◮ f ∗ is convex (no matter whether f is or not) ◮ proof: it is the pointwise supremum of a family of convex (indeed, affine) functions of y ◮ will be useful in Chapter 5 SJTU Ying Cui 27 / 42 Examples ◮ affine function f (x)= ax + b

b, y = a domf ∗ = a , f ∗(y) = sup(yx (ax+b)) = − { } x − ( , otherwise ∞ ◮ negative logarithm f (x)= log x with domf = R − ++ domf ∗ = R++, f ∗(y) =sup(yx + log x) − x>0 1 log( y), y < 0 = − − − ( , otherwise ∞ ◮ 1 T strictly convex quadratic function f (x)= 2 x Qx with Q Sn ∈ ++ n T 1 T 1 T 1 domf ∗ = R , f ∗(y) =sup(y x x Qx)= y Q− y x − 2 2

SJTU Ying Cui 28 / 42 Basic properties

◮ Fenchel’s inequality: f (x)+ f (y) xT y for all x, y ∗ ≥ ◮ conjugate of the conjugate: if f is convex and closed (i.e., epif is a closed set), then f ∗∗ = f ◮ differentiable functions: if f is convex and differentiable with domf = Rn, then f (y)= x T f (x ) f (x ), where x ∗ ∗ ∇ ∗ − ∗ ∗ satisfies y = f (x ) ∇ ∗ ◮ scaling and composition with affine transformation: ◮ for a > 0 and b R, the conjugate of g(x)= af (x)+ b is g ∗(y)= af ∗(y/a∈) b ◮ for nonsigular A −Rn×n and b Rn, the conjugate of g(x)= f (Ax + b∈) is g ∗(y)= f ∈∗(A−T y) bT A−T y with domg ∗ = AT domf − ◮ sum of independent functions: if f (u, v)= f1(u)+ f2(v), where f1 and f2 are convex functions with conjugates f1∗ and f2∗, respectively, then f ∗(w, z)= f1∗(w)+ f2∗(z)

SJTU Ying Cui 29 / 42 Quasiconvex functions f : Rn R is quasiconvex if domf is convex and all its sublevel → sets S = x domf f (x) α , a R are convex α { ∈ | ≤ } ∈

a b c

Figure 3.9 A quasiconvex function on R. For each α, the α-sublevel set Sα is convex, i.e. , an interval. The sublevel set Sα is the interval [a, b]. The sublevel set Sβ is the interval (−∞ , c].

◮ convex/quasiconvex functions have convex sublevel sets ◮ for a function on R, quasiconvexity requires that each sublevel set is an interval ◮ f is quasiconcave if f is quasiconvex, i.e., all its super level − sets x domf f (x) α , a R are convex { ∈ | ≥ } ∈ ◮ f is quasilinear if it is quasiconvex and quasiconcave, i.e., all its level sets x domf f (x)= α , a R are convex { ∈ | } ∈ SJTU Ying Cui 30 / 42 Examples

◮ x is quasiconvex on R ◮ | | pceil(x) = inf z Z z x is quasilinear ◮ { ∈ | ≥ } log x is quasilinear on R++ ◮ 2 f (x1, x2)= x1x2 is quasiconcave on R++ ◮ linear-fractional function aT x + b f (x)= , domf = x cT x + d > 0 cT x + d { | } is quasilinear ◮ distance ratio function x a f (x)= k − k2 , domf = x x a x b x b { | k − k2 ≤ k − k2} k − k2 is quasiconvex

SJTU Ying Cui 31 / 42 Basic properties A function f is quasiconvex iﬀ domf is convex and for any x, y domf and 0 θ 1, ∈ ≤ ≤ f (θx + (1 θ)y) max f (x), f (y) − ≤ { } which is called Jensen’s inequality for quasiconvex functions. ◮ the value of the function on a segment does not exceed the maximum of its values at the endpoints

max {f(x), f(y)} (y, f(y))

(x, f(x))

Figure 3.10 A quasiconvex function on R. The value of f between x and y is no more than max {f(x), f(y)}.

SJTU Ying Cui 32 / 42 Basic properties

Examples ◮ cardinality (number of nonzero components) of a nonnegative n n vector: card(x) is quasiconcave on R+ (but not R ), as card(x + y) min card(x), card(y) holds for x, y 0 ≥ { } ◮ rank of positive semideﬁnite matrix: rankX is quasiconcave on Sn , as rank(X + Y ) min rankX , rankY which holds + ≥ { } for X , Y Sn ∈ +

SJTU Ying Cui 33 / 42 Basic properties

◮ f is quasiconvex iﬀ its restriction to any line intersecting its domain is quasiconvex ◮ a continuous function f : R R is quasiconvex iﬀ at least one of the following conditions→ holds: ◮ f is nondecreasing ◮ f is nonincreasing ◮ there is a point c domf such that for t c (and t domf ), f is nonincreasing,∈ and for t ≤c (and t domf ), f ∈is nondecreasing, implying c is a global≥ minimizer of∈ f

c t

Figure 3.11 A quasiconvex function on R. The function is nonincreasing for t ≤ c and nondecreasing for t ≥ c.

SJTU Ying Cui 34 / 42 Differentiable quasiconvex functions First-order conditions Suppose f : Rn R is differentiable. → ◮ f is quasiconvex iff domf is convex and for all x, y domf , ∈ f (y) f (x) f (x)T (y x) 0 ≤ ⇒∇ − ≤ ◮ geometric interpretation: f (x) (when = 0) defines a supporting hyperplane to∇ sublevel set y6 f (y) f (x) , at x { | ≤ }

∇f(x) x

Figure 3.12 Three level curves of a quasiconvex function f are shown. The vector ∇f(x) deﬁnes a supporting hyperplane to the sublevel set {z | f(z) ≤ f(x)} at x.

SJTU Ying Cui 35 / 42 Differentiable quasiconvex functions Second-order conditions Suppose f : Rn R is twice differentiable. → ◮ if f is quasiconvex, then for all x domf and all y Rn, ∈ ∈ y T f (x) = 0 y T 2f (x)y 0 ∇ ⇒ ∇ ≥ ◮ when f (x) = 0, 2f (x) 0 ◮ when ∇f (x) = 0, ∇2f (x) is positive semidefinite on the (n 1)-dimensional∇ 6 ∇ subspace f (x)⊥, implying that 2f (x) can− have at most one negative∇ eigenvalue ∇ ◮ if f is quasiconvex on R, then for all x domf , ∈ f ′(x) = 0 f ′′(x) 0 ◮ at any⇒ point with≥ zero slope, the second derivative is nonnegative ◮ partial converse: if f satisfies y T f (x) = 0 y T 2f (x)y > 0 ∇ ⇒ ∇ for all x domf and all y Rn, y = 0, then f is quasiconvex ∈ ∈ 6 SJTU Ying Cui 36 / 42 Operations that preserve quasiconvexity Nonnegative weighted maximum ◮ if f , , fm are quasiconvex and w , , wm 0, then 1 · · · 1 · · · ≥ f = max w f , , wmfm is quasiconvex { 1 1 · · · } ◮ if g(x, y) is quasiconvex in x and w(y) 0, for all y C, ≥ ∈ then f (x) = supy C (w(y)g(x, y)) is quasiconvex ∈ Composition ◮ if g : Rn R is quasiconvex and h : R R is nondecreasing, → → then h(g(x)) is quasiconvex ◮ composition of a quasiconvex function with an affine or linear-fractional transformation yields a quasiconvex function Minimization ◮ if f (x, y) is quasiconvex jointly in x and y and C is a convex set, then the function g(x) = infy C f (x, y) is quasiconvex ∈

SJTU Ying Cui 37 / 42 Log-concave and log-convex functions ◮ a function f : Rn R is log-concave if f (x) > 0 for all → x domf and log f is concave ∈ ◮ a function f : Rn R is log-convex if f (x) > 0 for all → x domf and log f is convex ∈ ◮ f is log-convex iﬀ 1/f is log-concave ◮ a function f : Rn R with convex domain and f (x) > 0 for all x domf is log-concave→ iﬀ for all x, y domf and ∈ θ 1∈θ 0 θ 1, f (θx + (1 θ)y) f (x) f (y) − ≤◮ interpretation:≤ the− value at≥ the average of two points is at least the geometric mean of the values at the two points ◮ a log-convex function is convex ◮ composition rule: if g is convex, then eg is convex (g = log f ) ◮ a nonnegative concave function is log-concave ◮ composition rule: if g is concave, then log g is concave ◮ a log-convex (-concave) function is quasiconvex (-concave) ◮ any sublevel (superlevel) set of log f is convex and log is monotone increasing

SJTU Ying Cui 38 / 42 Examples

◮ aﬃne function: f (x)= aT x + b is log-concave on x aT x + b > 0 { | } ◮ powers: xa on R is log-convex for a 0, and log-concave ++ ≤ for a 0 ≥ ◮ exponentials: f (x) = exp(ax) is log-convex and log-concave ◮ cumulative Gaussian distribution function: 2 Φ(x)= 1 x e u /2du is log-concave √2π − −∞ ◮ x 1 u Gamma function:R Γ(x)= ∞ u − e− is log-convex for x 1 0 ≥ ◮ n determinant: detX is log-concaveR on S++ ◮ n determinant over trace: detX /trX is log-concave on S++ ◮ many common probability densities are log-concave, e.g., multivariate normal distribution, exponential distribution on n R+, and uniform distribution over a convex set

SJTU Ying Cui 39 / 42 Properties Twice differentiable log-convex/concave functions Suppose f is twice differentiable, with domf convex, so 1 1 2 log f (x)= 2f (x) f (x) f (x)T ∇ f (x)∇ − f (x)2 ∇ ∇ ◮ f is log-concave iff for all x domf , ∈ f (x) 2f (x) f (x) f (x)T ∇ ∇ ∇ ◮ f is log-convex iff for all x domf , ∈ f (x) 2f (x) f (x) f (x)T ∇ ∇ ∇ Multiplication, addition, and integration ◮ product of log-concave (log-convex) functions is log-concave (log-convex) ◮ positive scaling of log-concave (log-convex) functions is log-concave (log-convex) ◮ sum (integral) of log-concave functions is not always log-concave, but sum (integral) of log-convex functions is always log-convex SJTU Ying Cui 40 / 42 Consequences of integration property Integration of log-concave functions If f : Rn Rm R is log-concave, then g(x)= f (x, y)dy is × → log-concave in x on Rn (integration is over Rm) R ◮ marginal distributions of log-concave probability densities are log-concave ◮ convolution f g of log-concave functions f , g, ∗ (f g)(x)= f (x y)g(y)dy, is log-concave ◮∗ − proof: f (xR y) is log-concave in (x, y) (as f is concave), and product of− log-concave functions is log-concave ◮ if C Rn is convex and y is a random variable in Rn with log-concave⊆ p.d.f. p(y), then f (x)= prob(x + y C) is log-concave ∈ ◮ proof: write f (x) as integral of product of log-concave functions (which is still log-concave) 1 u C f (x)= g(x + y)p(y)dy, g(u)= ∈ ( 0 u / C R ∈ SJTU Ying Cui 41 / 42 Convexity with respect to generalized inequalities Suppose K Rm is a proper cone with associated generalized ⊆ inequality K ◮ f : Rn Rm is K-convex if domf is convex and →

f (θx + (1 θ)y) K θf (x) + (1 θ)f (y) − − for x, y domf and 0 θ 1 ∈ ≤ ≤ ◮ f : Rn Rm is strictly K-convex if domf is convex and →

f (θx + (1 θ)y) K θf (x) + (1 θ)f (y) − ≺ − for x, y domf , x = y and 0 θ 1 ∈ 6 ≤ ≤ ◮ these deﬁnitions reduce to ordinary convexity and strict convexity when m = 1 and K = R+. Many of the results for convex functions have extensions to K-convex functions.

SJTU Ying Cui 42 / 42