Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Math 408A Convexity
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
Convexity and optimality Directional Derivatives and Convexity
The Subdifferential Inequality
Convexity and Optimality Conditions
Sublinear Functionals
Examples of Convex Functions
Math 408A Convexity n Definition: A function f : R → R is said to be convex if for every n two points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2).
The function f is said to be strictly convex if for every two distinct n points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) < λf (x1) + (1 − λ)f (x2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
n Definition: A set C ⊂ R is said to be convex if for every x, y ∈ C and λ ∈ [0, 1] one has (1 − λ)x + λy ∈ C .
Math 408A Convexity The function f is said to be strictly convex if for every two distinct n points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) < λf (x1) + (1 − λ)f (x2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
n Definition: A set C ⊂ R is said to be convex if for every x, y ∈ C and λ ∈ [0, 1] one has (1 − λ)x + λy ∈ C .
n Definition: A function f : R → R is said to be convex if for every n two points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2).
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
n Definition: A set C ⊂ R is said to be convex if for every x, y ∈ C and λ ∈ [0, 1] one has (1 − λ)x + λy ∈ C .
n Definition: A function f : R → R is said to be convex if for every n two points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) ≤ λf (x1) + (1 − λ)f (x2).
The function f is said to be strictly convex if for every two distinct n points x1, x2 ∈ R and λ ∈ [0, 1] we have
f (λx1 + (1 − λ)x2) < λf (x1) + (1 − λ)f (x2).
Math 408A Convexity (x 2 , f (x 2 )) (x 1 , f (x 1 ))
x λ λ x 1 x 1 + (1 - )x 2 2 That is, the set
epi (f ) = {(x, µ): f (x) ≤ µ},
called the epi-graph of f , is a convex set.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
The secant line connecting (x1, f (x1)) and (x2, f (x2)) lies above the graph of f .
Math 408A Convexity That is, the set
epi (f ) = {(x, µ): f (x) ≤ µ},
called the epi-graph of f , is a convex set.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
The secant line connecting (x1, f (x1)) and (x2, f (x2)) lies above the graph of f .
(x 2 , f (x 2 )) (x 1 , f (x 1 ))
x λ λ x 1 x 1 + (1 - )x 2 2
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
The secant line connecting (x1, f (x1)) and (x2, f (x2)) lies above the graph of f .
(x 2 , f (x 2 )) (x 1 , f (x 1 ))
x λ λ x 1 x 1 + (1 - )x 2 2 That is, the set
epi (f ) = {(x, µ): f (x) ≤ µ},
called the epi-graph of f , is a convex set.
Math 408A Convexity Examples of convex functions are
cT x, kxk, xT Hx when H is psd.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
n Definition: A function f : R → R ∪ {+∞} = R¯ is said to be convex if the set epi (f ) = {(x, µ): f (x) ≤ µ} is a convex set. We also define the essential domain of f to be the set
dom (f ) = {x : f (x) < +∞} .
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity
n Definition: A function f : R → R ∪ {+∞} = R¯ is said to be convex if the set epi (f ) = {(x, µ): f (x) ≤ µ} is a convex set. We also define the essential domain of f to be the set
dom (f ) = {x : f (x) < +∞} .
Examples of convex functions are
cT x, kxk, xT Hx when H is psd.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality
n n Theorem: Let f : R → R¯ be convex. If x ∈ R is a local minimum for f , then x is a global minimum for f . If, in addition, f is strictly convex, then x¯ is the unique global minimizer.
Math 408A Convexity −1 Set λ := (2kx − bxk) < 1 and xλ := x + λ(bx − x). Then kxλ − xk ≤ /2 and
f (xλ) ≤ (1 − λ)f (x) + λf (bx) < f (x).
This contradicts the choice of and so no such bx exists.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality
Proof: Suppose f (bx) < f (x). Let > 0 be such that
f (x) ≤ f (x) whenever kx − xk ≤ and < 2kx − bxk .
Math 408A Convexity Then kxλ − xk ≤ /2 and
f (xλ) ≤ (1 − λ)f (x) + λf (bx) < f (x).
This contradicts the choice of and so no such bx exists.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality
Proof: Suppose f (bx) < f (x). Let > 0 be such that
f (x) ≤ f (x) whenever kx − xk ≤ and < 2kx − bxk . −1 Set λ := (2kx − bxk) < 1 and xλ := x + λ(bx − x).
Math 408A Convexity This contradicts the choice of and so no such bx exists.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality
Proof: Suppose f (bx) < f (x). Let > 0 be such that
f (x) ≤ f (x) whenever kx − xk ≤ and < 2kx − bxk . −1 Set λ := (2kx − bxk) < 1 and xλ := x + λ(bx − x). Then kxλ − xk ≤ /2 and
f (xλ) ≤ (1 − λ)f (x) + λf (bx) < f (x).
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality
Proof: Suppose f (bx) < f (x). Let > 0 be such that
f (x) ≤ f (x) whenever kx − xk ≤ and < 2kx − bxk . −1 Set λ := (2kx − bxk) < 1 and xλ := x + λ(bx − x). Then kxλ − xk ≤ /2 and
f (xλ) ≤ (1 − λ)f (x) + λf (bx) < f (x).
This contradicts the choice of and so no such bx exists.
Math 408A Convexity n 1. Given x, d ∈ R the difference quotient f (x + td) − f (x) t is a non-decreasing function of t on (0, +∞). n 0 2. For every x, d ∈ R the directional derivative f (x; d) always exists and is given by
f (x + td) − f (x) f 0(x; d) := inf . t>0 t
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
n Lemma: Let f : R → R¯ be convex (not necessarilty differentiable).
Math 408A Convexity n 0 2. For every x, d ∈ R the directional derivative f (x; d) always exists and is given by
f (x + td) − f (x) f 0(x; d) := inf . t>0 t
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
n Lemma: Let f : R → R¯ be convex (not necessarilty differentiable). n 1. Given x, d ∈ R the difference quotient f (x + td) − f (x) t is a non-decreasing function of t on (0, +∞).
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
n Lemma: Let f : R → R¯ be convex (not necessarilty differentiable). n 1. Given x, d ∈ R the difference quotient f (x + td) − f (x) t is a non-decreasing function of t on (0, +∞). n 0 2. For every x, d ∈ R the directional derivative f (x; d) always exists and is given by
f (x + td) − f (x) f 0(x; d) := inf . t>0 t
Math 408A Convexity Recall that f (x + td) − f (x) f 0(x; d) := lim . t↓0 t Now if the difference quotient on the right is non-decreasing in t on (0, +∞), then the limit is necessarily given by the infimum of these ratios. This infimum always exists and so f 0(x; d) always exists and is given by the infimum.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
We first assume (1) is true and show (2).
Math 408A Convexity This infimum always exists and so f 0(x; d) always exists and is given by the infimum.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
We first assume (1) is true and show (2). Recall that f (x + td) − f (x) f 0(x; d) := lim . t↓0 t Now if the difference quotient on the right is non-decreasing in t on (0, +∞), then the limit is necessarily given by the infimum of these ratios.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
We first assume (1) is true and show (2). Recall that f (x + td) − f (x) f 0(x; d) := lim . t↓0 t Now if the difference quotient on the right is non-decreasing in t on (0, +∞), then the limit is necessarily given by the infimum of these ratios. This infimum always exists and so f 0(x; d) always exists and is given by the infimum.
Math 408A Convexity n Let x, d ∈ R and let 0 < t1 < t2. Then t1 f (x + t1d) = f x + t t2d h 2 i t1 t1 = f 1 − t x + t (x + t2d) 2 2 t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0.
Math 408A Convexity t1 f (x + t1d) = f x + t t2d h 2 i t1 t1 = f 1 − t x + t (x + t2d) 2 2 t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0. n Let x, d ∈ R and let 0 < t1 < t2. Then
Math 408A Convexity h i t1 t1 = f 1 − t x + t (x + t2d) 2 2 t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0. n Let x, d ∈ R and let 0 < t1 < t2. Then t1 f (x + t1d) = f x + t2d t2
Math 408A Convexity t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0. n Let x, d ∈ R and let 0 < t1 < t2. Then t1 f (x + t1d) = f x + t t2d h 2 i t1 t1 = f 1 − x + (x + t2d) t2 t2
Math 408A Convexity Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0. n Let x, d ∈ R and let 0 < t1 < t2. Then t1 f (x + t1d) = f x + t t2d h 2 i t1 t1 = f 1 − t x + t (x + t2d) 2 2 t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives and Convexity Directional Derivatives and Convexity
f (x+td)−f (x) We now show that t is non-decreasing for t > 0. n Let x, d ∈ R and let 0 < t1 < t2. Then t1 f (x + t1d) = f x + t t2d h 2 i t1 t1 = f 1 − t x + t (x + t2d) 2 2 t1 t1 ≤ 1 − f (x) + f (x + t2d). t2 t2
Hence f (x + t d) − f (x) f (x + t d) − f (x) 1 ≤ 2 . t1 t2
Math 408A Convexity Plug in t = 1 on the right hand side to get
0 n f (y) ≥ f (x) + f (x;(y − x)) ∀ y ∈ R .
WeWhen willf callis differentiable this inequality at thex, thissubdifferential inequality becomes inequality. T n f (y) ≥ f (x) + ∇f (x) (y − x) ∀ y ∈ R .
This inequality states that the tangent plane to the graph of f lies entirely below the graph of f .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
The Subdifferential Inequality
n n If f : R → R is convex, then for all x ∈ dom (f ) and y ∈ R , f (x + t(y − x)) − f (x) f 0(x;(y − x)) = inf . t>0 t
Math 408A Convexity WeWhen willf callis differentiable this inequality at thex, thissubdifferential inequality becomes inequality. T n f (y) ≥ f (x) + ∇f (x) (y − x) ∀ y ∈ R .
This inequality states that the tangent plane to the graph of f lies entirely below the graph of f .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
The Subdifferential Inequality
n n If f : R → R is convex, then for all x ∈ dom (f ) and y ∈ R , f (x + t(y − x)) − f (x) f 0(x;(y − x)) = inf . t>0 t Plug in t = 1 on the right hand side to get
0 n f (y) ≥ f (x) + f (x;(y − x)) ∀ y ∈ R .
Math 408A Convexity When f is differentiable at x, this inequality becomes
T n f (y) ≥ f (x) + ∇f (x) (y − x) ∀ y ∈ R .
This inequality states that the tangent plane to the graph of f lies entirely below the graph of f .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
The Subdifferential Inequality
n n If f : R → R is convex, then for all x ∈ dom (f ) and y ∈ R , f (x + t(y − x)) − f (x) f 0(x;(y − x)) = inf . t>0 t Plug in t = 1 on the right hand side to get
0 n f (y) ≥ f (x) + f (x;(y − x)) ∀ y ∈ R .
We will call this inequality the subdifferential inequality.
Math 408A Convexity We will call this inequality the subdifferential inequality.
This inequality states that the tangent plane to the graph of f lies entirely below the graph of f .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
The Subdifferential Inequality
n n If f : R → R is convex, then for all x ∈ dom (f ) and y ∈ R , f (x + t(y − x)) − f (x) f 0(x;(y − x)) = inf . t>0 t Plug in t = 1 on the right hand side to get
0 n f (y) ≥ f (x) + f (x;(y − x)) ∀ y ∈ R .
When f is differentiable at x, this inequality becomes
T n f (y) ≥ f (x) + ∇f (x) (y − x) ∀ y ∈ R .
Math 408A Convexity We will call this inequality the subdifferential inequality.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
The Subdifferential Inequality
n n If f : R → R is convex, then for all x ∈ dom (f ) and y ∈ R , f (x + t(y − x)) − f (x) f 0(x;(y − x)) = inf . t>0 t Plug in t = 1 on the right hand side to get
0 n f (y) ≥ f (x) + f (x;(y − x)) ∀ y ∈ R .
When f is differentiable at x, this inequality becomes
T n f (y) ≥ f (x) + ∇f (x) (y − x) ∀ y ∈ R .
This inequality states that the tangent plane to the graph of f lies entirely below the graph of f .
Math 408A Convexity (i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
Math 408A Convexity 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x).
Math 408A Convexity (iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
Math 408A Convexity Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x).
Math 408A Convexity (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done.
Math 408A Convexity (iii)⇒(i): Obvious.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n Theorem: Let f : R → R¯ be convex (not necessarilty differentiable) and let x¯ ∈ dom (f ). Then the following three statements are equivalent.
(i)¯x is a local solution to minx∈Rn f (x). 0 n (ii) f (¯x; d) ≥ 0 for all d ∈ R .
(iii)¯x is a global solution to minx∈Rn f (x). Proof: (i)⇒(ii): Done. (ii)⇒(iii): Follows from the subdifferential inequality;
0 n f (y) ≥ f (¯x) + f (¯x;(y − x)) ≥ f (¯x) ∀ y ∈ R .
(iii)⇒(i): Obvious.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Convexity and Optimality Conditions
n n Theorem: Let f : R → R be convex and suppose that x ∈ R is a point at which f is differentiable. Then x is a global minimum of f if and only if ∇f (x) = 0.
Math 408A Convexity (1) f is convex, (2) f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ Rn (3)( ∇f (x) − ∇f (y))T (x − y) ≥ 0 for all x, y ∈ Rn. If f is twice differentiable then f is convex if and only if 2 n ∇ f (x) is positive semi-definite for all x ∈ R .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking Convexity
n Theorem: Let f : R → R¯. n If f is differentiable on R , then the following statements are equivalent.
Math 408A Convexity (2) f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ Rn (3)( ∇f (x) − ∇f (y))T (x − y) ≥ 0 for all x, y ∈ Rn. If f is twice differentiable then f is convex if and only if 2 n ∇ f (x) is positive semi-definite for all x ∈ R .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking Convexity
n Theorem: Let f : R → R¯. n If f is differentiable on R , then the following statements are equivalent. (1) f is convex,
Math 408A Convexity (3)( ∇f (x) − ∇f (y))T (x − y) ≥ 0 for all x, y ∈ Rn. If f is twice differentiable then f is convex if and only if 2 n ∇ f (x) is positive semi-definite for all x ∈ R .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking Convexity
n Theorem: Let f : R → R¯. n If f is differentiable on R , then the following statements are equivalent. (1) f is convex, (2) f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ Rn
Math 408A Convexity If f is twice differentiable then f is convex if and only if 2 n ∇ f (x) is positive semi-definite for all x ∈ R .
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking Convexity
n Theorem: Let f : R → R¯. n If f is differentiable on R , then the following statements are equivalent. (1) f is convex, (2) f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ Rn (3)( ∇f (x) − ∇f (y))T (x − y) ≥ 0 for all x, y ∈ Rn.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking Convexity
n Theorem: Let f : R → R¯. n If f is differentiable on R , then the following statements are equivalent. (1) f is convex, (2) f (y) ≥ f (x) + ∇f (x)T (y − x) for all x, y ∈ Rn (3)( ∇f (x) − ∇f (y))T (x − y) ≥ 0 for all x, y ∈ Rn. If f is twice differentiable then f is convex if and only if 2 n ∇ f (x) is positive semi-definite for all x ∈ R .
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ (∇f (x) − ∇f (y))T (x − y) ≥ 0 ∀ x, y ∈ Rn
n For all x, y ∈ R , the subdifferential inequality gives
f (y) ≥ f (x) + ∇f (x)T (y − x) f (x) ≥ f (y) + ∇f (y)T (x − y)
Adding these two inequalities and canceling like terms from each side gives 0 ≥ ∇f (x)T (y − x) + ∇f (y)T (x − y) or equivalently
0 ≤ (∇f (x) − ∇f (y))T (x − y) .
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
(∇f (x) − ∇f (y))T (x − y) ≥ 0 ∀ x, y ∈ Rn ⇒ f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn
n Let x, y ∈ R . By the MVT there exists 0 < λ < 1 such that
T f (y) − f (x) = ∇f (xλ) (y − x)
where xλ := λy + (1 − λ)x. Then
T 0 ≤ [∇f (xλ) − ∇f (x)] (xλ − x) T = λ[∇f (xλ) − ∇f (x)] (y − x) = λ[f (y) − f (x) − ∇f (x)T (y − x)].
Hence f (y) ≥ f (x) + ∇f (x)T (y − x).
Math 408A Convexity Show α ≤ 0. [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α. If λ equals 0 or 1, we are done, so assume 0 < λ < 1. Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ). Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1]
Math 408A Convexity [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α. If λ equals 0 or 1, we are done, so assume 0 < λ < 1. Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ). Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1] Show α ≤ 0.
Math 408A Convexity If λ equals 0 or 1, we are done, so assume 0 < λ < 1. Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ). Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1] Show α ≤ 0. [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α.
Math 408A Convexity Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ). Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1] Show α ≤ 0. [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α. If λ equals 0 or 1, we are done, so assume 0 < λ < 1.
Math 408A Convexity Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1] Show α ≤ 0. [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α. If λ equals 0 or 1, we are done, so assume 0 < λ < 1. Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ).
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
f (y) ≥ f (x) + ∇f (x)T (y − x) ∀x, y ∈ Rn ⇒ f is convex
Let x, y ∈ Rn and set α := max ϕ(λ) := [f (λy + (1 − λ)x) − (λf (y) + (1 − λ)f (x))]. λ∈[0,1] Show α ≤ 0. [0, 1] is compact and ϕ continuous, so ∃ λ ∈ [0, 1] such that ϕ(λ) = α. If λ equals 0 or 1, we are done, so assume 0 < λ < 1. Then 0 T 0 = ϕ (λ) = ∇f (xλ) (y − x) + f (x) − f (y)
where xλ = x + λ(y − x), or equivalently T λ(f (y) − f (x)) = −∇f (xλ) (x − xλ). Hence, by the subdifferential inequality,
α = f (xλ) − (f (x) + λ(f (y) − f (x))) T = f (xλ) + ∇f (xλ) (x − xλ) − f (x) ≤ 0
Math 408A Convexity So t2 f (x) + t∇f (x)T d + dT ∇2f (x)d + o(t2) ≥ f (x) + t∇f (x)T d 2 or equivalently 1 o(t2) dT ∇2f (x)d + ≥ 0. 2 t2 Letting t → 0 yields the inequality dT ∇2f (x)d ≥ 0.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇒ ∇2f (x) PSD on Rn If f is convex, T n f (x + td) ≥ f (x) + t∇f (x) d ∀ t ∈ R, x, d ∈ R .
Math 408A Convexity or equivalently 1 o(t2) dT ∇2f (x)d + ≥ 0. 2 t2 Letting t → 0 yields the inequality dT ∇2f (x)d ≥ 0.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇒ ∇2f (x) PSD on Rn If f is convex, T n f (x + td) ≥ f (x) + t∇f (x) d ∀ t ∈ R, x, d ∈ R . So t2 f (x) + t∇f (x)T d + dT ∇2f (x)d + o(t2) ≥ f (x) + t∇f (x)T d 2
Math 408A Convexity Letting t → 0 yields the inequality dT ∇2f (x)d ≥ 0.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇒ ∇2f (x) PSD on Rn If f is convex, T n f (x + td) ≥ f (x) + t∇f (x) d ∀ t ∈ R, x, d ∈ R . So t2 f (x) + t∇f (x)T d + dT ∇2f (x)d + o(t2) ≥ f (x) + t∇f (x)T d 2 or equivalently 1 o(t2) dT ∇2f (x)d + ≥ 0. 2 t2
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇒ ∇2f (x) PSD on Rn If f is convex, T n f (x + td) ≥ f (x) + t∇f (x) d ∀ t ∈ R, x, d ∈ R . So t2 f (x) + t∇f (x)T d + dT ∇2f (x)d + o(t2) ≥ f (x) + t∇f (x)T d 2 or equivalently 1 o(t2) dT ∇2f (x)d + ≥ 0. 2 t2 Letting t → 0 yields the inequality dT ∇2f (x)d ≥ 0.
Math 408A Convexity Hence f (y) ≥ f (x) + ∇f (x)T (y − x) 2 since ∇ f (xλ) is positive semi-definite. Therefore, f is convex.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇐ ∇2f (x) PSD on Rn
n For x, y ∈ R , the MVT implies there exists λ ∈ (0, 1) such that 1 f (y) = f (x) + ∇f (x)T (y − x) + (y − x)T ∇2f (x )(y − x) 2 λ
where xλ = λy + (1 − λ)x .
Math 408A Convexity Therefore, f is convex.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇐ ∇2f (x) PSD on Rn
n For x, y ∈ R , the MVT implies there exists λ ∈ (0, 1) such that 1 f (y) = f (x) + ∇f (x)T (y − x) + (y − x)T ∇2f (x )(y − x) 2 λ
where xλ = λy + (1 − λ)x . Hence f (y) ≥ f (x) + ∇f (x)T (y − x) 2 since ∇ f (xλ) is positive semi-definite.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions f convex ⇐ ∇2f (x) PSD on Rn
n For x, y ∈ R , the MVT implies there exists λ ∈ (0, 1) such that 1 f (y) = f (x) + ∇f (x)T (y − x) + (y − x)T ∇2f (x )(y − x) 2 λ
where xλ = λy + (1 − λ)x . Hence f (y) ≥ f (x) + ∇f (x)T (y − x) 2 since ∇ f (xλ) is positive semi-definite. Therefore, f is convex.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking for Positive Definiteness
n×n Theorem: Let H ∈ R be symmetric. We define the kth principal minor of H, denoted ∆k (H), to be the determinant of the upper left-hand k × k submatrix of H. Then H is positive definite if and only if ∆k (H) > 0 for k = 1, 2,..., n.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Checking for Positive Definiteness
Example: 1 1 −1 H = 1 5 1 . −1 1 4 We have
1 1 ∆1(H) = 1, ∆2(H) = = 4, ∆3(H) = det(H) = 8. 1 5
Therefore, H is positive definite.
Math 408A Convexity Note: Sublinear functionals can be thought of as a generalization of the notion of a norm. In particular, every norm is a sublinear functional.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
n Definition: Let h : R → R ∪ {+∞}. We say that h is positively homogeneous if
h(λx) = λh(x) for all x ∈ R and λ > 0.
We say that h is subadditive if
h(x + y) ≤ h(x) + h(y) for all x, y ∈ R.
Finally, we say that h is sublinear if it is both subadditive and positively homogeneous.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
n Definition: Let h : R → R ∪ {+∞}. We say that h is positively homogeneous if
h(λx) = λh(x) for all x ∈ R and λ > 0.
We say that h is subadditive if
h(x + y) ≤ h(x) + h(y) for all x, y ∈ R.
Finally, we say that h is sublinear if it is both subadditive and positively homogeneous. Note: Sublinear functionals can be thought of as a generalization of the notion of a norm. In particular, every norm is a sublinear functional.
Math 408A Convexity Lemma: Every sublinear functional is a convex function.
Proof:
h(λx + (1 − λ)y) ≤ h(λx) + h(1 − λ)y) = λh(x) + (1 − λ)h(y).
Lemma: A function is sublinear if and only if its epigraph is a convex cone.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
Sublinear functionals form a very rich and important class of functionals with a surprisingly beautiful structure.
Math 408A Convexity Proof:
h(λx + (1 − λ)y) ≤ h(λx) + h(1 − λ)y) = λh(x) + (1 − λ)h(y).
Lemma: A function is sublinear if and only if its epigraph is a convex cone.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
Sublinear functionals form a very rich and important class of functionals with a surprisingly beautiful structure.
Lemma: Every sublinear functional is a convex function.
Math 408A Convexity Lemma: A function is sublinear if and only if its epigraph is a convex cone.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
Sublinear functionals form a very rich and important class of functionals with a surprisingly beautiful structure.
Lemma: Every sublinear functional is a convex function.
Proof:
h(λx + (1 − λ)y) ≤ h(λx) + h(1 − λ)y) = λh(x) + (1 − λ)h(y).
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Sublinear Functionals
Sublinear functionals form a very rich and important class of functionals with a surprisingly beautiful structure.
Lemma: Every sublinear functional is a convex function.
Proof:
h(λx + (1 − λ)y) ≤ h(λx) + h(1 − λ)y) = λh(x) + (1 − λ)h(y).
Lemma: A function is sublinear if and only if its epigraph is a convex cone.
Math 408A Convexity Then at every point x ∈ dom (f ) the directional derivative f 0(x; d) is a sublinear function of the d argument.
0 n That is, the function f (x; ·): R → R ∪ {+∞} is sublinear.
Thus, in particular, the function f 0(x; ·) is a convex function.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Theorem: Let f : R → R ∪ {+∞} be a convex function.
Math 408A Convexity 0 n That is, the function f (x; ·): R → R ∪ {+∞} is sublinear.
Thus, in particular, the function f 0(x; ·) is a convex function.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Theorem: Let f : R → R ∪ {+∞} be a convex function.
Then at every point x ∈ dom (f ) the directional derivative f 0(x; d) is a sublinear function of the d argument.
Math 408A Convexity Thus, in particular, the function f 0(x; ·) is a convex function.
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Theorem: Let f : R → R ∪ {+∞} be a convex function.
Then at every point x ∈ dom (f ) the directional derivative f 0(x; d) is a sublinear function of the d argument.
0 n That is, the function f (x; ·): R → R ∪ {+∞} is sublinear.
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Theorem: Let f : R → R ∪ {+∞} be a convex function.
Then at every point x ∈ dom (f ) the directional derivative f 0(x; d) is a sublinear function of the d argument.
0 n That is, the function f (x; ·): R → R ∪ {+∞} is sublinear.
Thus, in particular, the function f 0(x; ·) is a convex function.
Math 408A Convexity f (x + tλd) − f (x) = lim λ t↓0 λt f (x + (tλ)d) − f (x) = λ lim (λt)↓0 (λt) = λf 0(x; d).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Proof: (Positive Homogeneity) Let x ∈ dom (f ), d ∈ R , and λ > 0. Then f (x + tλd) − f (x) f 0(x; λd) = lim t↓0 t
Math 408A Convexity f (x + (tλ)d) − f (x) = λ lim (λt)↓0 (λt) = λf 0(x; d).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Proof: (Positive Homogeneity) Let x ∈ dom (f ), d ∈ R , and λ > 0. Then f (x + tλd) − f (x) f 0(x; λd) = lim t↓0 t f (x + tλd) − f (x) = lim λ t↓0 λt
Math 408A Convexity = λf 0(x; d).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Proof: (Positive Homogeneity) Let x ∈ dom (f ), d ∈ R , and λ > 0. Then f (x + tλd) − f (x) f 0(x; λd) = lim t↓0 t f (x + tλd) − f (x) = lim λ t↓0 λt f (x + (tλ)d) − f (x) = λ lim (λt)↓0 (λt)
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n Proof: (Positive Homogeneity) Let x ∈ dom (f ), d ∈ R , and λ > 0. Then f (x + tλd) − f (x) f 0(x; λd) = lim t↓0 t f (x + tλd) − f (x) = lim λ t↓0 λt f (x + (tλ)d) − f (x) = λ lim (λt)↓0 (λt) = λf 0(x; d).
Math 408A Convexity f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t 0 0 = f (x; d1) + f (x; d2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t
Math 408A Convexity 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t 0 0 = f (x; d1) + f (x; d2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t
Math 408A Convexity 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t 0 0 = f (x; d1) + f (x; d2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t
Math 408A Convexity f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t 0 0 = f (x; d1) + f (x; d2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t
Math 408A Convexity 0 0 = f (x; d1) + f (x; d2).
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Directional Derivatives of Convex Functions are Sublinear
n (Subadditivity) let d1, d2 ∈ R , Then
0 f (x + t(d1 + d2)) − f (x) f (x; d1 + d2) = lim t↓0 t f ( 1 (x + 2td ) + 1 (x + 2td )) − f (x) = lim 2 1 2 2 t↓0 t 1 f (x + 2td ) + 1 f (x + 2td ) − f (x) ≤ lim 2 1 2 2 t↓0 t 1 (f (x + 2td ) − f (x)) + 1 (f (x + 2td ) − f (x)) ≤ lim 2 1 2 2 t↓0 t f (x + 2td ) − f (x) f (x + 2td ) − f (x) = lim 1 + lim 2 t↓0 2t t↓0 2t 0 0 = f (x; d1) + f (x; d2).
Math 408A Convexity f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
Math 408A Convexity f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
Math 408A Convexity 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn}
Math 408A Convexity f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++
Math 408A Convexity 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn )
Math 408A Convexity n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1
Math 408A Convexity n f (x) = − log det(H)) H ∈ S++
Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++
Math 408A Convexity Outline Convexity Convexity and optimality The Subdifferential Inequality Convexity and Optimality Conditions Sublinear Functionals Examples of Convex Functions
Examples of Convex Functions
n n n Set R++ = {x ∈ R |0 < xi , i = 1,..., n } and S++ equal to the set on n × n symmetric positive definite matrices.
f (x) = kxk
f (x) = max{x1, x2,..., xn} 1 f (x) = x T Hx + g T x + α H ∈ n 2 S++ f (x) = log(ex1 + ex2 + ··· + exn ) 1/n n Y n f (x) = − xi x ∈ R++ j=1 n f (x) = − log(x1x2 ... xn) x ∈ R++ n f (x) = − log det(H)) H ∈ S++
Math 408A Convexity