Linear Algebra Review

CSE 203B: Convex Optimization Week 4 Discuss Session

1 Contents

• Convex functions (Ref. Chap.3) • Review: definition, first order condition, second order condition, operations that preserve convexity • Epigraph • Conjugate function • Dual norm

2 Review of Convex Function

• Definition ( often simplified by restricting to a line) • First order condition: a global underestimator • Second order condition • Operations that preserve convexity • Nonnegative weighted sum • Composition with affine function • Pointwise maximum and supremum • Composition • Minimization See Chap 3.2, try to prove why the convexity • Perspective is preserved with those operations.

3 Epigraph • 훼-sublevel set of 푓: 푅푛 → 푅 퐶훼 = 푥 ∈ 푑표푚 푓 푓 푥 ≤ 훼} sublevel sets of a convex function are convex for any value of 훼. • Epigraph of 푓: 푅푛 → 푅 is defined as 퐞퐩퐢 푓 = (푥, 푡) 푥 ∈ 푑표푚 푓, 푓 푥 ≤ 푡} ⊆ 푅푛+1

• A function is convex iff its epigraph is a convex set. 4 Relation between convex sets and convex functions • A function is convex iff its epigraph is a convex set. • Consider a convex function 푓 and 푥, 푦 ∈ 푑표푚 푓 푡 ≥ 푓 푦 ≥ 푓 푥 + 훻푓 푥 푇(푦 − 푥)

epi 풇 First order condition for convexity • The hyperplane supports epi 풇 at (푥, 푓 푥 ), for any 푡 푥 푦, 푡 ∈ 퐞퐩퐢 푓 ⇒ 훻푓 푥 푇 푦 − 푥 + 푓 푥 − 푡 ≤ 0 훻푓 푥 푇 푦 푥 ⇒ − ≤ 0 −1 푡 푓 푥 Supporting hyperplane, derived from first order condition

5 Pointwise Supremum • If for each 푦 ∈ 푈, 푓(푥, 푦): 푅푛 → 푅 is convex in 푥, then function 푔(푥) = sup 푓(푥, 푦) 푦∈푈 is convex in 푥. • Epigraph of 푔 푥 is the intersection of epigraphs with 푓 and set 푈 퐞퐩퐢 푔 =∩푦∈푈 퐞퐩퐢 푓(∙ , 푦) knowing 퐞퐩퐢 푓(∙ , 푦) is a convex set (푓 is a convex function in 푥 and regard 푦 as a const), so 퐞퐩퐢 푔 is convex. • An interesting method to establish convexity of a function: the pointwise supremum of a family of affine functions. (ref. Chap 3.2.4)

6 Conjugate Function • Given function 푓: 푅푛 → 푅 , the conjugate function 푓∗ 푦 = sup 푦푇푥 − 푓(푥) 푥∈푑표푚 푓 • The dom 푓∗ consists 푦 ∈ 푅푛 for which sup 푦푇푥 − 푓(푥) is bounded 푥∈푑표푚 푓

Supporting hyperplane: if 푓 is convex in that domain Slope 푦 = −1 (푥෤, 푓(푥෤)) where 훻푓푇 푥෤ = 푦 if 푓 is differentiable

Slope 푦 = 0 (0, −푓∗(0)) ∗ (Proof with pointwise supremum) Theorem: 푓 (푦) is convex even 푓 푥 is not convex. 푻 풚 풙 − 풇(풙) is affine function7 in 풚 Examples of Conjugates • Derive the conjugates of 푓: 푅 → 푅 푓∗ 푦 = sup 푦푥 − 푓(푥) 푥∈푑표푚 푓 Affine

quadratic Norm See the extension to domain 푹풏 in Example 3.21

8 Examples of Conjugates

n 푥푖 푛 • Log-sum-exp: 푓 푥 = log(Σ푖=1푒 ) , 푥 ∈ 푅 . The conjugate is 푓∗ 푦 = sup 푦푇푥 − 푓(푥) 푥∈푑표푚 푓 Since 푓(푥) is differentiable, first calculate the gradient of 푔 푥 = 푦푇푥 − 푓(푥) 푍 훻푔 푥 = 푦 − ퟏ푇푍 where 푍 = 푒푥1, ⋯ 푒푥푛 푇. The maximum of 푔 푥 is attained at 훻푔 푥 = 0, 푍 so that 푦 = . ퟏ푇푍 푒푥푖 푦 = , 푖 = 1, … , 푛 푖 n 푥푖 Σ푖=1푒 Does the bounded supremum (not → ∞) exist for all 풚 ∈ 푹풏 ?

9 Examples of Conjugates Does the bounded supremum (not → ∞) exist for all 풚 ∈ 푹풏 ? 푒푥푖 • If 푦 < 0, then the gradient on 훻 푔 = 푦 − < 0 푥 ∈ 푅. 푖 푖 푖 n 푥푖 for all 푖 Then Σ푖=1푒 푔 푥 reaches maximum at ∞ at 푥 → −∞. So 푦 ≽ 0. 푍 푍 • To have 푦 = solvable, we notice that ퟏ푇푦 = ퟏ푇 = 1. 1푇푍 1푇푍 If ퟏ푇푦 ≠ 1 and 푦 ≽ 0, we choose 푥 = 푡ퟏ s.t. 푔 푥 = 푦푇푥 − 푓 푥 = 푡푦푇ퟏ − log 푛푒푡 = 푡 푦푇ퟏ − 1 − log 푛 푔 푥 reaches maximum at ∞ when we let 푡 → ∞(−∞), which is not bounded. Σn 푦 log 푦 if ퟏ푇푦 = 1 and 푦 ≽ 0 • The conjugate function 푓∗ 푦 = ቊ 푖=1 푖 푖 ∞ otherwise

10 Dual Norm

n • Suppose ∥·∥ is a norm on R , the dual norm ∥·∥∗ is defined as 푇 푛 푛 • ∥ 푧 ∥∗ = sup 푧 푥: 푥 ∈ 푅 , ∥ 푥 ∥ ≤ 1 , ∀푧 ∈ 푅

• We will show that the dual norm of the 푙푝 norm is the 푙푞 norm • With Holder’s inequality:

• We have Holder’s inequality

• Hence the supremum is at most ∥ 푧 ∥푝 11 Dual Norm

• We need to show that the supremum is exactly ∥ 푧 ∥푝, we need to 푛 푛 푇 find a single 푦∈ ℝ such that σ푖=1 푧 y = ∥ 푧 ∥푝

푥 • We can choose 푦 = , to prove the equality holds. ∥푥∥푞 • More details: https://math.stackexchange.com/questions/265721/proving- that-the-dual-of-the-mathcall-p-norm-is-the-mathcall-q-norm