Convex Functions Prof. Daniel P. Palomar ELEC5470/IEDA6100A - Convex Optimization The Hong Kong University of Science and Technology (HKUST) Fall 2020-21 Outline of Lecture • Definition convex function • Examples on R, Rn, and Rn×n • Restriction of a convex function to a line • First- and second-order conditions • Operations that preserve convexity • Quasi-convexity, log-convexity, and convexity w.r.t. generalized inequalities (Acknowledgement to Stephen Boyd for material for this lecture.) Daniel P. Palomar 1 Definition of Convex Function • A function f : Rn −! R is said to be convex if the domain, dom f, is convex and for any x; y 2 dom f and 0 ≤ θ ≤ 1, f (θx + (1 − θ) y) ≤ θf (x) + (1 − θ) f (y) : • f is strictly convex if the inequality is strict for 0 < θ < 1. • f is concave if −f is convex. Daniel P. Palomar 2 Examples on R Convex functions: • affine: ax + b on R • powers of absolute value: jxjp on R, for p ≥ 1 (e.g., jxj) p 2 • powers: x on R++, for p ≥ 1 or p ≤ 0 (e.g., x ) • exponential: eax on R • negative entropy: x log x on R++ Concave functions: • affine: ax + b on R p • powers: x on R++, for 0 ≤ p ≤ 1 • logarithm: log x on R++ Daniel P. Palomar 3 Examples on Rn • Affine functions f (x) = aT x + b are convex and concave on Rn. n • Norms kxk are convex on R (e.g., kxk1, kxk1, kxk2). • Quadratic functions f (x) = xT P x + 2qT x + r are convex Rn if and only if P 0. Qn 1=n n • The geometric mean f (x) = ( i=1 xi) is concave on R++. P xi n • The log-sum-exp f (x) = log i e is convex on R (it can be used to approximate max xi). i=1;:::;n 2 n • Quadratic over linear: f (x; y) = x =y is convex on R × R++. Daniel P. Palomar 4 Examples on Rn×n • Affine functions: (prove it!) f (X) = Tr (AX) + b are convex and concave on Rn×n. • Logarithmic determinant function: (prove it!) f (X) = logdet (X) is concave on Sn = fX 2 Rn×n j X 0g. • Maximum eigenvalue function: (prove it!) yT Xy f (X) = λmax (X) = sup T y6=0 y y is convex on Sn. Daniel P. Palomar 5 Epigraph • The epigraph of f if the set epi f = (x; t) 2 Rn+1 j x 2 dom f; f (x) ≤ t : • Relation between convexity in sets and convexity in functions: f is convex () epi f is convex Daniel P. Palomar 6 Restriction of a Convex Function to a Line • f : Rn −! R is convex if and only if the function g : R −! R g (t) = f (x + tv) ; dom g = ft j x + tv 2 dom fg is convex for any x 2 dom f; v 2 Rn. • In words: a function is convex if and only if it is convex when restricted to an arbitrary line. • Implication: we can check convexity of f by checking convexity of functions of one variable! • Example: concavity of logdet (X) follows from concavity of log (x). Daniel P. Palomar 7 Example: concavity of logdet (X) : g (t) = logdet (X + tV ) = logdet (X) + logdet I + tX−1=2VX−1=2 n X = logdet (X) + log (1 + tλi) i=1 −1=2 −1=2 where λi's are the eigenvalues of X VX . The function g is concave in t for any choice of X 0 and V ; therefore, f is concave. Daniel P. Palomar 8 First and Second Order Condition • Gradient (for differentiable f): h iT rf (x) = @f(x) ··· @f(x) 2 Rn: @x1 @xn • Hessian (for twice differentiable f): @2f (x) r2f (x) = 2 Rn×n: @xi@xj ij • Taylor series: 1 f (x + δ) = f (x) + rf (x)T δ + δT r2f (x) δ + o kδk2 : 2 Daniel P. Palomar 9 • First-order condition: a differentiable f with convex domain is convex if and only if f (y) ≥ f (x) + rf (x)T (y − x) 8x; y 2 dom f • Interpretation: first-order approximation if a global underestimator. • Second-order condition: a twice differentiable f with convex domain is convex if and only if r2f (x) 0 8x 2 dom f Daniel P. Palomar 10 Examples • Quadratic function: f (x) = (1=2) xT P x + qT x + r (with P 2 Sn) rf (x) = P x + q; r2f (x) = P is convex if P 0. 2 • Least-squares objective: f (x) = kAx − bk2 rf (x) = 2AT (Ax − b) ; r2f (x) = 2AT A is convex. Daniel P. Palomar 11 • Quadratic-over-linear: f (x; y) = x2=y 2 y r2f (x; y) = y −x 0 y3 −x is convex for y > 0. Daniel P. Palomar 12 Operations that Preserve Convexity How do we establish the convexity of a given function? 1. Applying the definition. 2. With first- or second-order conditions. 3. By restricting to a line. 4. Showing that the functions can be obtained from simple functions by operations that preserve convexity: • nonnegative weighted sum • composition with affine function (and other compositions) • pointwise maximum and supremum, minimization • perspective Daniel P. Palomar 13 • Nonnegative weighted sum: if f1; f2 are convex, then α1f1 +α2f2 is convex, with α1; α2 ≥ 0. • Composition with affine functions: if f is convex, then f (Ax + b) is convex (e.g., ky − Axk is convex, logdet I + HXHT is con- cave). • Pointwise maximum: if f1; : : : ; fm are convex, then f (x) = max ff1; : : : ; fmg is convex. n Example: sum of r largest components of x 2 R : f (x) = x[1] + x[2] + ··· + x[r] where x[i] is the ith largest component of x. Proof: f (x) = max fxi1 + xi2 + ··· + xir j 1 ≤ i1 < i2 < ··· < ir ≤ ng. Daniel P. Palomar 14 • Pointwise supremum: if f (x; y) is convex in x for each y 2 A, then g (x) = sup f (x; y) y2A is convex. Example: distance to farthest point in a set C: f (x) = sup kx − yk : y2C Example: maximum eigenvalue of symmetric matrix: for X 2 Sn, yT Xy λmax (X) = sup T : y6=0 y y Daniel P. Palomar 15 • Composition with scalar functions: let g : Rn −! R and h : R −! R, then the function f (x) = h (g (x)) satisfies: g convex; h convex nondecreasing f (x) is convex if g concave; h convex nonincreasing • Minimization: if f (x; y) is convex in (x; y) and C is a convex set, then g (x) = inf f (x; y) y2C is convex (e.g., distance to a convex set). Example: distance to a set C: f (x) = inf kx − yk y2C is convex if C is convex. Daniel P. Palomar 16 • Perspective: if f (x) is convex, then its perspective g (x; t) = tf (x=t) ; dom g = (x; t) 2 Rn+1 j x=t 2 dom f; t > 0 is convex. Example: f (x) = xT x is convex; hence g (x; t) = xT x=t is convex for t > 0. Example: the negative logarithm f (x) = − log x is convex; hence the 2 relative entropy function g (x; t) = t log t − t log x is convex on R++. Daniel P. Palomar 17 Quasi-Convexity Functions • A function f : Rn −! R is quasi-convex if dom f is convex and the sublevel sets Sα = fx 2 dom f j f (x) ≤ αg are convex for all α. • f is quasiconcave if −f is quasiconvex. Daniel P. Palomar 18 Examples: • pjxj is quasiconvex on R • ceil (x) = inf fz 2 Z j z ≥ xg is quasilinear • log x is quasilinear on R++ 2 • f (x1; x2) = x1x2 is quasiconcave on R++ • the linear-fractional function aT x + b f (x) = ; dom f = x j cT x + d > 0 cT x + d is quasilinear Daniel P. Palomar 19 Log-Convexity • A positive function f is log-concave is log f is concave: f (θx + (1 − θ) y) ≥ f (x)θ f (y)1−θ for 0 ≤ θ ≤ 1: • f is log-convex if log f is convex. a • Example: x on R++ is log-convex for a ≤ 0 and log-concave for a ≥ 0 • Example: many common probability densities are log-concave Daniel P. Palomar 20 Convexity w.r.t. Generalized Inequalities • f : Rn −! Rm is K-convex if dom f is convex and for any x; y 2 dom f and 0 ≤ θ ≤ 1, f (θx + (1 − θ) y) K θf (x) + (1 − θ) f (y) : m m 2 m • Example: f : S −! S , f (X) = X is S+ -convex. Daniel P. Palomar 21 References Chapter 3 of • Stephen Boyd and Lieven Vandenberghe, Convex Optimization. Cambridge, U.K.: Cambridge University Press, 2004. https://web.stanford.edu/~boyd/cvxbook/bv cvxbook.pdf Daniel P. Palomar 22.