Convex functions I ELL 822–Selected Topics in Communications Let f : S → R, where S is a nonempty convex set in Rn. The function f is convex on S if f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2) for each x1, x2 ∈ S and for each λ ∈ (0, 1) Lecture 2 Convex functions - Ref: [Boyd] Chapter 3 Strictly convex on S if the above inequality is true as a strict inequality Jun B. Seo 2-1 2-2 Level sets Sublevel sets • Sublevel set associated with f is defined as (for α ∈ R) Level set of function f with level α is defined as Sα = {x ∈ dom f |f (x) ≤ α} S = {x|f (x) = α} • Let S be a nonempty convex set in Rn and let f : S → R be a – Styblinski-Tang function: f (x) = 0.5(Pn x4 − 16x2 + 5x ) i=1 i i i convex function. Then, the sublevel set Sα is convex: Proof Suppose x , x ∈ S . Thus, we have x , x ∈ S and 4 1 2 α 1 2 3 100 f (x1) ≤ α and f (x2) ≤ α. 2 50 1 Consider x = λx1 + (1 − λ)x2 for λ ∈ (0, 1). 0 2 0 x -50 -1 By convexity of S, we can see x ∈ S and x ∈ Sα. -2 -100 4 -3 Since f is convex, we write 2 4 0 2 -4 x2 -2 0 -2 x1 -4 -4 -4 -2 0 2 4 x1 f (x) ≤ λf (x1) + (1 − λ)f (x2) ≤ λα + (1 − λ)α = α. 2-3 2-4 Epigraph I Epigraph II • Let S be a nonempty set in Rn and let f : S → R. Proof • The graph of f is described by the set {(x, f (x))|x ∈ S} ⊂ Rn+1 Suppose that f is convex and let (x1, y1), and (x2, y2) ∈ epi f • The epigraph of f , denoted by epi f , is a subset of Rn+1, i.e., – It means that x1, x2 ∈ S and y1 ≥ f (x1) and y2 ≥ f (x2). epi f = {(x, y)|x ∈ S, y ∈ R, y ≥ f (x)} – Convexity of f enables us to write f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2) ≤ λy1 + (1 − λ)y2 – Since λx1 + (1 − λ)x2 ∈ S, we have [x1 + (1 − λ)x2, λy1 + (1 − λ)y2] ∈ epi f • Let S be a nonempty convex set. Then, f is convex if and only if epi f is a convex set: 2-5 2-6 Epigraph III Epigraph IV Constrained optimization problem in standard form Proof continued minimize f0(x) x Conversely assume that epi f is convex and let x1, x2 ∈ S subject to fi (x) ≤ 0, for i = 1,..., m Then, while [x1, f (x1)] ∈ epi f and [x2, f (x2)] ∈ epi f , due to hj (x) = 0, for j = 1,..., p convexity of epi f , for λ ∈ (0, 1) we have We can rewrite this in epigraph from as [λx + (1 − λ)x , λf (x ) + (1 − λ)f (x )] ∈ epi f . 1 2 1 2 minimize t x,t It implies subject to f0(x) − t ≤ 0 f (x) ≤ 0, for i = 1,..., m f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2), i hj (x) = 0, for j = 1,..., p which is convex. – Every convex optimization problem can be transformed into a problem with linear objective function 2-7 2-8 Epigraph V First-order condition of convex functions I The epigraph form is an optimization problem in the (epi)graph Let S be a nonempty open set in Rn and let f : S → R be space (x, t): differentiable on S. minimize t Then, f is convex if and only if for any x ∈ S, we have x,t min. f0(x) = |x| x T ⇒ subject to |x| − t ≤ 0 f (y) ≥ f (x) + ∇f (x) (y − x) for each y ∈ S, subject to − x + 1 ≤ 0 − x + 1 ≤ 0 h df (x) df (x) iT where ∇f (x) = ,..., is gradient of f . dx1 dxn 3 feasible region -3 0 1 3 – The first order approximation of f at x is a global lower bound 2-9 2-10 First-order condition of convex functions II First-order condition of convex functions III Proof Proof continued By convexity of f , To show the converse, suppose a point t = αx + (1 − α)y f (αy + (1 − α)x) ≤ αf (y) + (1 − α)f (x) We need to show that f is convex if the following holds = α(f (y) − f (x)) + f (x) f (x) ≥ f (t) + ∇f (t)(x − t) We rewrite this as f (y) ≥ f (t) + ∇f (t)(y − t) f (αy + (1 − α)x) − f (x) + αf (x) ≤ αf (y). Multiplying each with α and 1 − α, we have Finally, we have αf (x) ≥ αf (t) + α∇f (t)(x − t) f (x + α(y − x)) − f (x) (1 − α)f (y) ≥ (1 − α)f (t) + (1 − α)∇f (t)(y − t) f (y) ≥f (x) + (y − x) α(y − x) Adding them yields f (x + ∆x) − f (x) =f (x) + (y − x) αf (x) + (1 − α)f (y) ≥ f (t) + ∇f (t)(αx + (1 − α)y − t) ∆x =f (x) + ∇f (x)T (y − x) = f (αx + (1 − α)y) 2-11 2-12 First-order condition of convex functions IV Convex functions II • If f is convex and x, y ∈ dom f , we have Let S be an nonempty convex set in Rn and let f : S → R be t ≥ f (y) ≥ f (x) + ∇f (x)T (y − x) for (y, t) ∈ epi f differentiable on S. • The epi f has a supporting hyperplane with [∇f (x), −1] at x Then, f is convex if and only if for each x1, x2 ∈ S, we have " # " #! y x [∇f (x ) − ∇f (x )]T (x − x ) ≥ 0 (monotone) (y, t) ∈ epi f ⇒ [∇f (x) − 1] − ≤ 0 2 1 2 1 t f (x) Proof If f is convex, for two distinct x1 and x2 we have T f (x1) ≥ f (x2) + ∇f (x2) (x1 − x2) T f (x2) ≥ f (x1) + ∇f (x1) (x2 − x1) Adding two equations side-by-side, we have T T ∇f (x2) (x1 − x2) + ∇f (x1) (x2 − x1) ≤ 0 Global minimum of convex function f is attained if and only if non-vertical supporting ∇f (x) = 0 hyperplanes 2-13 2-14 Convex functions III Second-order condition of convex functions I Proof continued • Let Sn denote the set of symmetric n × n matrices, i.e., To prove the converse, by assumption, if the following holds for Sn = {X ∈ Rn×n|X = X T } x = λx1 + (1 − λ)x2 – Sn (or, Sn ) denote the set of symmetric positive [∇f (x) − ∇f (x )]T (x − x ) ≥ 0, ++ + 1 1 (semi)definite matrix, i.e., zT H z > 0 (or, zT H z ≥ 0) for z ∈ Rn T we have (1 − λ)[∇f (x) − ∇f (x1)] (x2 − x1) ≥ 0, i.e., • Let S be a nonempty open set in Rn and let f : S → R be twice T T differentiable on S. ∇f (x) (x2 − x1) ≥ ∇f (x1) (x2 − x1) • f is convex (strictly) function if and only if its Hessian matrix Using the mean value theorem, i.e., ∂2f (x) T H (x) = [hij (x)] with hij (x) = f (x2) − f (x1) = ∇f (x) (x2 − x1), ∂xi ∂xj H (x) ∈ Sn (or Sn ) over S for x = λx1 + (1 − λ)x2 and λ ∈ (0, 1). We have the result. + ++ 2-15 2-16 Second-order condition of convex functions II Second-order condition of convex functions III Proof Proof continued Using convexity of f , f (y) ≥ f (x) + ∇f (x)(y − x) for To show the converse, use ‘mean value theorem’ extended to y = x + λx ∈ S with small λ, we have second order f (x + λx) ≥ f (x) + λ∇f (x)T x Let f : Rn → R be twice continuously differentiable over an open set S, and x ∈ S. Using Taylor expansion of f , we also have For all y such that x + y ∈ S there exists an α ∈ [0, 1] 1 f (x + λx) = f (x) + λ∇f (x)T x + λ2xT H (x)x + λ2kxk2O(x; λx) 2 1 f (x + y) = f (x) + yT ∇f (x) + yT ∇2f (x + αy)y where O(x; λx) → 0 as λ → 0. 2 Plugging this, dividing λ2 and letting λ → 0 yields In using mean value theorem, let x = x + y ∈ S. Then, 1 1 T f (x) = f (x) + yT ∇f (x) + yT ∇2f (x + αy)y x H (x)x + O(x; λx) ≥ 0 2 2 | {z } →0 2-17 2-18 Second-order condition of convex functions IV Restriction of a convex function to a line I • f : Rn → R is convex if and only if g : R → R Proof continued g(t) = f (x + tv) for dom g = {t|x + tv ∈ domf } The point x + αy in ∇2f (x + αy) is expressed as is convex (in t) for any x ∈ dom f , v ∈ Rn x + αy = x + α(x − x) = αx + (1 − α)x = xˆ : used to check convexity of f by checking convexity of functions of one variable The theorem gives f x , x x2 x2 1 ( 1 2) = 1 + 2 T T 2 f x , x x2 − x2 f (x) = f (x) + y ∇f (x) + y ∇ f (xˆ)y ( 1 2) = 1 2 2 60 20 1 T 2 40 0 If 2 y ∇ f (xˆ)y ≥ 0, then -20 20 f (x) ≥ f (x) + ∇f (x)T (x − x), -40 0 4 6 2 4 4 2 6 0 2 which completes the proof 0 2 4 x2 -2 0 x -2 -4 -2 0 -4 -2 x 2 -6 -6 -4 x1 -4 1 2-19 2-20 Restriction of a convex function to a line II Restriction of a convex function to a line III Proof n n f : S → R and f (X) = log det X for dom f = S++.