Chapter 5 Convex Optimization in Function Space 5.1 Foundations of Convex Analysis Let V be a vector space over lR and k ¢ k : V ! lR be a norm on V . We recall that (V; k ¢ k) is called a Banach space, if it is complete, i.e., if any Cauchy sequence fvkglN of elements vk 2 V; k 2 lN; converges to an element v 2 V (kvk ¡ vk ! 0 as k ! 1). Examples: Let ­ be a domain in lRd; d 2 lN. Then, the space C(­) of continuous functions on ­ is a Banach space with the norm kukC(­) := sup ju(x)j : x2­ The spaces Lp(­); 1 · p < 1; of (in the Lebesgue sense) p-integrable functions are Banach spaces with the norms Z ³ ´1=p p kukLp(­) := ju(x)j dx : ­ The space L1(­) of essentially bounded functions on ­ is a Banach space with the norm kukL1(­) := ess sup ju(x)j : x2­ The (topologically and algebraically) dual space V ¤ is the space of all bounded linear functionals ¹ : V ! lR. Given ¹ 2 V ¤, for ¹(v) we often write h¹; vi with h¢; ¢i denoting the dual product between V ¤ and V . We note that V ¤ is a Banach space equipped with the norm j h¹; vi j k¹k := sup : v2V nf0g kvk Examples: The dual of C(­) is the space M(­) of Radon measures ¹ with Z h¹; vi := v d¹ ; v 2 C(­) : ­ The dual of L1(­) is the space L1(­). The dual of Lp(­); 1 < p < 1; is the space Lq(­) with q being conjugate to p, i.e., 1=p + 1=q = 1. The dual of L1(­) is the space of Borel measures. A Banach space V is said to be reflexive, if V ¤¤ = V . In view of the examples before, the spaces Lp(­); 1 < p < 1; are reflexive, but C(­) and L1(­);L1(­) are not. 95 96 Optimization I; Chapter 5 We denote by 2V ¤ the power set of V ¤, i.e., the set of all subsets of V ¤. De¯nition 5.1 (Weighted duality mapping) Let h : lR+ ! lR+ be a continuous and non-decreasing function such that h(0) = 0 and h(t) ! +1 as t ! +1. Then, the mapping ¤ ¤ ¤ ¤ Jh(u) := fu 2 V j hu; u i = kukku k ; ku k = h(kuk)g is called the weighted (or gauged) duality mapping, and h is referred to as the weight (or gauge). The weighted duality mapping is surjective, if and only if V is reflexive. Example: For V = Lp(­);V ¤ = Lq(­); 1 < p; q < +1; 1=p + 1=q = 1; and h(t) = tp¡1, we have ½ ju(x)jp¡1sgn(u(x)) ; u(x) 6= 0 J (u)(x) := : h 0 ; u(x) = 0 Let V be a Banach space and uk 2 V; k 2 lN; and u 2 V . The sequence fukglN is said to converge strongly to u ( uk ! u (k ! 1) or s-lim uk = u), if kuk ¡ uk ! 0 (k ! 1). The sequence fukglN is said to converge weakly to u ( uk * u (k ! 1) ¤ or w-lim uk = u), if h¹; uk ¡ ui ! 0 (k ! 1) for all ¹ 2 V . Theorem 5.2 (Theorem of Eberlein/Shmulyan) In a reflexive Banach space V a bounded sequence fukglN; uk 2 V; k 2 lN; contains a weakly convergent subsequence, i.e., there exist a subse- 0 0 quence lN ½ lN and an element u 2 V such that uk * u (k 2 lN ! 1). In the sequel, we assume V to be a reflexive Banach space. De¯nition 5.3 (Convex set, convex hull) Let u; v 2 V . By [u; v] ½ V we denote the line-segment with endpoints u and v according to [u; v] := f¸u + (1 ¡ ¸)v j ¸ 2 [0; 1]g : A set A ½ V is called convex, if and only if for any u; v 2 A the segment [u; v] is contained in A as well. The convex hull co A of a subset A ½ V is the convex combination of all elements of A, i.e., Xn Xn co A := f ¸iui j n 2 lN; ¸i = 1; ¸i ¸ 0; ui 2 A; 1 · i · ng : i=1 i=1 Optimization I; Chapter 5 97 The closure of the convex hull co A is said to be the closed convex hull. De¯nition 5.4 (A±ne hyperplane, supporting hyperplane, separation of sets) Let ¹ 2 V ¤; ¹ 6= 0; and ® 2 lR. The set of elements H := fv 2 V j ¹(v) = ®g is called an a±ne hyperplane. The convex sets fv 2 V j ¹(v) < ®g ; fv 2 V j ¹(v) · ®g ; fv 2 V j ¹(v) > ®g ; fv 2 V j ¹(v) ¸ ®g are called open resp. closed half-spaces bounded by H. If A ½ V and H is a closed, a±ne hyperplane containing at least one point u 2 A such that A is completely contained in one of the closed half-spaces determined by H, then H is called a supporting hyperplane of A and u is said to be a supporting point of A. An a±ne hyperplane H is said to separate (strictly separate) two sets A; B ½ V , if each of the closed (open) half-spaces bounded by H con- tains one of them, i.e., ¹(u) · ® ; u 2 A ; ¹(v) ¸ ® ; v 2 B resp. ¹(u) < ® ; u 2 A ; ¹(v) > ® ; v 2 B: Theorem 5.5 (Geometrical form of the Hahn-Banach theo- rem) Let A ½ V be an open, non-empty, convex set and M a non-empty a±ne subspace with A \ M = ;. Then, there exists a closed a±ne hyperplane H with M ½ H and A \H = ;. Corollary 5.6 (Separation of convex sets) (i) Let A ½ V be an open, non-empty, convex set and B ½ V a non- empty, convex set with A \ B = ;. Then, there exists a closed a±ne hyperplane H which separates A and B. (ii) Let A ½ V be a compact, non-empty convex set and B ½ V a closed, non-empty, convex set with A \ B = ;. Then, there exists a closed a±ne hyperplane H which strictly separates A and B. A consequence of Corollary 5.6 (i) is: Corollary 5.7 (Boundary of convex sets) Let A ½ V be a convex set with non-empty interior. Then, any bound- ary point of A is a supporting point of A. As a consequence of Corollary 5.6 (ii) we obtain: 98 Optimization I; Chapter 5 Corollary 5.8 (Characterization of closed convex sets) Any closed convex set A ½ V is the intersection of the closed half- spaces which contain A. In particular, every closed convex set is weakly closed. The converse of Corollary 5.8 is known as Mazur's lemma: Lemma 5.9 (Mazur's Lemma) Let fukglN; uk 2 V; k 2 lN; and u 2 V such that w-lim uk = u. Then, there is a sequence fvkglN of convex combinations XK XK vk = ¸iui ; ¸i = 1 ; ¸i ¸ 0 ; k · i · K; i=k i=k such that s-lim vk = u. The combination of Corollary 5.8 and Lemma 5.9 gives: Corollary 5.10 (Properties of convex sets) A convex set A ½ V is closed if and only if it is weakly closed. De¯nition 5.11 (Convex function, strictly convex function, e®ective domain) Let A ½ V be a convex set and f : A ! lR := [¡1; +1]. Then, f is said to be convex if for any u; v 2 A there holds f(¸u + (1 ¡ ¸)v) · ¸f(u) + (1 ¡ ¸)f(v) ; ¸ 2 [0; 1] : A function f : A ! lR is called strictly convex if f(¸u + (1 ¡ ¸)v) < ¸f(u) + (1 ¡ ¸)f(v) ; ¸ 2 (0; 1) : A function f : A ! lR is called proper convex if f(u) > ¡1; u 2 A; and f 6´ +1. If f : A ! lR is convex, the convex set dom f := fu 2 A j f(u) < +1g is called the e®ective domain of f. De¯nition 5.12 (Indicator function) If A ½ V , the indicator function ÂA of A is de¯ned by means of ½ 0 ; u 2 V  (u) := : A +1 ; u2 = V The indicator function of a convex set A is a convex function. Optimization I; Chapter 5 99 De¯nition 5.13 (Epigraph of a function) Let f : V ! lR be a function. The set epi f := f(u; a) 2 V £ lR j f(u) · ag is called the epigraph of f. The projection of epi f onto V is the e®ective domain dom f. Theorem 5.14 (Characterization of convex functions) A function f : V ! lR is convex if and only if its epigraph is convex. Proof: Let f be convex and assume (u; a); (v; b) 2 epi f. Then, for all ¸ 2 [0; 1] f(¸u + (1 ¡ ¸)v) · ¸f(u) + (1 ¡ ¸)f(v) · ¸a + (1 ¡ ¸)b ; and hence, ¸(u; a) + (1 ¡ ¸)(v; b) 2 epi f. Conversely, assume that epi f is convex. It su±ces to verify the con- vexity of f on its e®ective domain. For that purpose, let u; v 2 dom f such that a ¸ f(u) and b ¸ f(v). Since ¸(u; a) + (1 ¡ ¸)(v; b) 2 epi f for every ¸ 2 [0; 1] it follows that f(¸u + (1 ¡ ¸)v) · ¸a + (1 ¡ ¸)b : If both f(u) and f(v) are ¯nite, we choose a = f(u) and b = f(v). If f(u) = ¡1 or f(v) = ¡1, it su±ces to allow a ! ¡1 resp.