Can. J. Math., Vol. XXXVIII, No. 2, 1986, pp. 431-452

PROXIMAL ANALYSIS AND BOUNDARIES OF CLOSED SETS IN , PART I: THEORY

J. M. BORWEIN AND H. M. STROJWAS

0. Introduction. As various types of tangent cones, generalized deriva­ tives and subgradients prove to be a useful tool in nonsmooth optimization and nonsmooth analysis, we witness a considerable interest in analysis of their properties, relations and applications. Recently, Treiman [18] proved that the Clarke tangent cone at a point to a closed subset of a Banach space contains the limit inferior of the contingent cones to the set at neighbouring points. We provide a considerable strengthening of this result for reflexive spaces. Exploring the analogous inclusion in which the contingent cones are replaced by pseudocontingent cones we have observed that it does not hold any longer in a general Banach space, however it does in reflexive spaces. Among the several basic relations we have discovered is the following one: the Clarke tangent cone at a point to a closed subset of a reflexive Banach space is equal to the limit inferior of the weak (pseudo) contingent cones to the set at neighbouring points. This generalizes the results of Penot [14] and Cornet [7] for finite dimensional spaces and of Penot [14] for reflexive spaces. What is more we show that this equality characterizes reflex­ ive spaces among Banach spaces, similarly as the analogous equality with the contingent cones instead of the weak contingent cones, characterizes finite dimensional spaces among Banach spaces. We also present several other related characterizations of reflexive spaces and variants of the considered inclusions for boundedly relatively weakly compact sets. For several years it has been an open question as to how to extend Clarke's proximal normal formula [6] from finite dimensional spaces to infinite dimension. We give the answer to this question for reflexive Banach spaces. We present a characterization of the Clarke normal cone at a point to a closed subset of a reflexive Banach space by means of the proximal normal functionals and the other one, by means of the Frechet normals. The importance of our result is underlined by the fact that we use exact normals in contrast to the c-normals used for example in [18]. Our main results are presented in the first part of the paper, the second part is devoted to their applications. For example, we obtain generaliza-

Received November 8, 1984. The research of the first author was partially supported on NSERC Grant A5116 while that of the second was partially supported on National Science Foundation Grant ECS 8300214.

431

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 432 J. M. BORWEIN AND H. M. STROJWAS

tions of results given in [12] and of the primal Bishop-Phelps theorem proven in [1]. We complete the results of Borwein and O'Brien [4] and of Borwein [2] on pseudoconvexity. We mention implications of our main results for vector fields and give detailed analysis of consequences in the theory of differentiability and subdifferentiability. We show how the "lim inf" inclusions under consideration relate to the subdifferential properties of spaces and functions. In particular we show that every lower semicontinuous function in a reflexive Banach space is densely sub- Frechet and that every locally Lipschitian function in a weakly compactly generated Banach space is densely sub-Hadamard. We prove the proximal subgradient formula in a reflexive Banach space which extends one for a finite dimensional space given in [17]. We also provide many examples which are carefully analyzed.

1. Preliminaries. Let E be a normed space with continuous linear dual E*. For the subset C c E and a point x G C we will consider the following tangent cones to C at x:

1) the contingent cone Kc(x) and

2) the Clarke tangent cone Tc(x), as defined for example in [6], where also the corresponding references may be found; 3) the weak (sequential) contingent cone WKc(x), which is the set of weak limits of sequences t~\cn — x), with tn > 0, tn —> 0, and cn G C; 4) the weak (sequential) tangent cone WTc(x\ which we define to be the set of those y G E, that for any sequences xn and tn such that xn G C, tn > 0, xn —> x, tn —> 0 there exists a sequence cn9 cn G C for which the sequence t~ (cn — xn) tends weakly to 7; 5) the pseudocontingent cone Pc(x) := ~cdKc(x)\ 6) the weak pseudocontingent cone WPc(x) := ~côWKc(x)\ 1) the Bony tangent cone, defined as the set of vectors y G E such that whenever n is a proximal normal vector to C at x, then the inequality ll« - ty\\ iï NI holds for every / > 0. We recall the definition of a proximal normal vector. Definition 1.1. Vector n G E is said to be a proximal normal vector to C at x G C if there are u <£ C and X > 0 such that

n = X(u — x) and \\u — x\\ = dc(u), where

dc(u) : = inf \\u — c||.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 433

As a straightforward consequence of the introduced definitions we obtain the following inclusions:

WTc(x) c WKc{x) c WPc(x)

(1.1) U U U

Tc(x) c Kc{x) c pc(x).

PROPOSITION 1.1. Bc(x) is a closed cone which satisfies the inclusion

(1.2) Kc{x) c Bc(x). Furthermore it is convex, whenever E is a smooth Banach space in which case also

(1.3) Pc(x) c Bc(x). If the of E is Frechet differentiable (that is Frechet differentiable away from zero), then

(1.4) WPc(x) c Bc(x). Proof of the above proposition follows from [14]. Inclusion (1.4) is implied by the one obtained in [14] for another type of the weak contingent cone: its definition uses bounded nets instead of sequences. It may be observed that in a these two definitions are equivalent. In all the sequel, whenever we refer to a general approximating cone which is among the defined above, we will denote it Rc(x). Definition 1.2. We say that C is R-pseudoconvex at x e C if

(1.5) C - x c Rc(x). If (1.5) holds for all x G C, C is R-pseudoconvex.

Definition 1.3. We say x e C is an R-proper point of C if Rc(x) ¥= E. Inclusions (1.1)-(1.4) imply obvious relations for .R-pseudoconvexity and .R-properness. We will consider also the polars of the approximating cones:

R°(x) : = {y* G E*\ (y*9 y) ^ 0 for all y G RC(X) }.

Nc(x) := Tç(x) is called the Clarke normal cone. We point out that the pseudocontingent cone and the weak pseudocon­ tingent cone are essential for dual and differentiability results. The following theorem was first proven in [18]. Alternative proofs may be also found in [15], [16] and in [5].

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 434 J. M. BORWEIN AND H. M. STROJWAS

THEOREM 1.1 (Treiman): If C is a closed subset of a Banach space, x G C, then

lim inf Kc(x') c Tc(x). x'—*x C

THEOREM 1.2 (Penot). 7/*2s is reflexive C ci E, x ΠC then

Tc(x) c lim inf WKc(x'). x'^x C The proof of Theorem 1.2 can be found in [14].

2. Tangent cone separation principle and stars of closed sets in Banach spaces. Definition 2.1. star C := {x e C| [je, j] c C for all 7 G C}, where [x, y] := co{x, j}. We say C is starshaped when star C is non-empty.

PROPOSITION 2.1. For any subset C of a {locally convex vector) space E the following inclusion holds

(2.1) star C c n Tc(x) + x.

/Vex?/. Let x e star C and JC G C. Let V be any neighbourhood of 0. Then for any x' G (JC + V) Pi C and any /, 0 < / ^ 1, we have

t_1[(f(jt - jcr) + jcr) - xr] H- F c r\c - y) + v,

which shows that x — x e TC(JC), hence (2.1) follows. In particular Proposition 2.1 implies that convex sets are 7"-pseudocon- vex in any locally convex . For closed sets in Banach spaces the inclusion in (2.1) may be replaced with equality. This is a consequence of the following theorem.

THEOREM 2.1. Let C be a closed subset of a Banach space E. Suppose that x in C and y in E are such that

(2.2) [JC, y] t C.

Then for any r > 0 one can find xr in C with

(i) y £ Kc(xr) + xr and

(ii) \\xr - x\\ ^ 117 - x\\ + r.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 435

Proof. (2.2) implies that there is some z in [x, y] and some s > 0 such that (z + sB) n C = 0, where 5 is a closed unit ball in E. Take r e (0, s). Applying Ekeland's variational principle as given in [9], to the function z •-> Ik - 711 on the set D := C n U _ (z 4 r£), ze[jc,z]

yields an jcr in Z) such that for all x in D

(2.3) ||x - y|| ^ IK - yll - q\\x - xr\\, for some q e (0, 1). Furthermore there exist w > 0 and /? > 0 such that

(2.4) xr 4- (0, w)(j - xr + /?£) c U__ (z + rB). z<^[x,z] But (2.3) shows that for some^ > 0 the "drop" ( [16] )

xr 4 (0, w)(y - xr + pB) does not intersect D, which together with (2.4) and the definition of D implies that it also does not intersect C. Thus

y - xr £ Kc(xr), which proves (i), and (ii) follows from the choice of D.

COROLLARY 2.1. If C is a closed subset of a Banach space then

(2.5) star C = n Tc(x) + x = n X"c(x) 4- JC,

#«d when C is starshaped

(2.6) rec(star C) = n Tc(x) = n ^c(x),

where rec(star C) w f/ze recession cone of star C. Proof By Theorem 2.1

n #C(JC) 4 JC c star C.

With Proposition 2.1 this gives (2.5). Since Tc(x) is convex for all x we have

[ n TC(X)] + [rc(x) 4 JC] C TC(JC) + JC,

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 436 J. M. BORWEIN AND H. M. STROJWAS

and therefore

n Tc(x) c rec(star C). XGC Conversely, suppose that h is in rec(star C), that is star C + R+/* c star C. Then for y in star C and x e C we have by (2.1)

y + nh G Tc(x) + * for all « e N\{0}, or

_1 _1 h G ^2 rc(x) + tf (* - 7),

which since Tc(x) is a closed cone shows that h G TC(X). Thus

rec(star C) c n Tc(x),

and by Theorem 1.1,

n rc(x) = n #c(*),

which finishes the proof. In the following sections we will present a strengthening of Corollary 2.1 for boundedly relatively weakly compact sets and we will show its applications.

3. "Lim inf" inclusions and proximal normal formulas in a reflexive Banach space. Let C be a closed subset of a Banach space E, x G C. Definition 3.1. A functional n* e E* is said to be a proximal normal functional to C at x if there are u £ C, X > 0, and y* e E* such that

(3.1) /1* = Ay*, ||w - JC|| = £/C(K), and (3.2) (y*, u - x) = \\u - x\\, \\y*\\ = 1, that is 7* supports unit ball at \\u — x\\~ (u — x), which is the proximal normal vector to C at x. The set of all proximal normal functionals to C at

x will be denoted PNC(JC). Definition 3.2. A functional/* e £* is said to be a Frechet normal to C at x if for any e > 0 there exists a neighbourhood X€ of JC such that the inequality

(3.3) (/*, JC' - JC) - CHJC' - JC|| ë 0

holds for all x' G C n X€.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 437

PROPOSITION 3.1. The set of all Frechet normals to C at x is included in

WK°c(x) and it coincides with WK°c(x) whenever E is reflexive.

Proof. Let/* be a Frechet normal to C at x and y G WKC(X). Then ] there exists a sequence Ç (cn ~ x) converging weakly to y with tn > 0, tn —> 0 and cn G C. Take any e > 0. Since weak convergent sequences are norm bounded

] t~ \\cn - x\\ ^ M for n G N and some M > 0.

Furthermore for sufficiently large n all cn lie in Xe from Definition 3.2. Hence by (3.3) (f*,y) = lim CV/*,^ - x) ^€M,

which, since € can be arbitrarily small, implies (/*, y) ^ 0. Therefore

/* G WK°c(x). Suppose now that E is reflexive,/* G WK°C(X) but (3.3) does not hold. Then there exists e > 0 and a sequence c„, c„ G C, cn —» x for which

(3.4) (/*, c„ - x) > c||cw - x\\. x Let j be a weak limit point of \\cn — x\\~ (cn — x). Then^ G WKc(x) and (3.4) shows (/*, y) ^ c, which is a contradiction. Hence we conclude that /* is a Frechet normal and the proof is finished.

PROPOSITION 3.2. If E is a smooth space then

(3.5) Bc(x) = PN£(JC).

Proof. Observe that whenever E is smooth, then A?* G PNC(X), ||«*|| = 1 implies that n* is the of the norm at some proximal normal vector to C at x. Furthermore any Gateaux derivative of the norm at the proximal normal vector to C at x is in PNc(x). Propositions 1.1, 3.1, and 3.2 imply the following.

COROLLARY 3.1. If E is reflexive and the norm of E is Frechet differentiable, then for any C c E, x G C, any proximal normal functional to C at x is a Frechet normal to C at x. Equipped with the definitions and basic relations of normal functionals we take on the task of extending the proximal normal formula

(3.6) Nc(x) = cô{jy* G E*\y*= lim j*, xn —» x, y*

is a proximal normal functional to C at xn } from the Euclidean space to a reflexive space. The following two propositions concerning the properties of reflexive spaces will be helpful on our way.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 438 J. M. BORWEIN AND H. M. STROJWAS

PROPOSITION 3.3. Every reflexive Banach space can be given an equivalent Frechet differentiable and locally uniformly convex (hence Kadec) norm. The proof is given in [8], where also the corresponding definitions may be found. Recall that a Kadec norm is one for which the weak and norm topologies agree on the boundary of the .

PROPOSITION 3.4. If E is a reflexive Banach space with a Kadec norm and C is a closed subset of E, then the set of those points which have a nearest point in C is dense in E. The above proposition follows from Lau's nearest point result proven in [13].

LEMMA 1. Let E be a Banach space with a closed unit ball B. For a > 0 and y G E define

(3.7) Wa := co{.y + aB, -y + aB, B).

Let || || and \\\\a denote the norms on E associated with B and Wa, respectively. Then

(i) if a > 0 and E is reflexive, or if a ^ 1 then the ball Wa is closed, the norms \\\\ and \\\\a are equivalent and the norm \\ \\a is smooth when­ ever the norm \\ \\ is smooth', (ii) if a > 0 and E is reflexive or if a i=! 1, then the norm \\ \\a is Kadec whenever the norm \\ \\ is Kadec, (iii) if a è 1, or if a > 0 and E is reflexive and the norm \\ \\ is Kadec, then the norm \\\\a is Frechet differ entiable whenever the norm \\\\ is Frechet differ entiable.

Proof. Let us observe that Wa of (3.7) may be written as

(3.8) Wa = co([-l, \]y + aB,B), and

(3.9) Wa = [-1, \]y + aB if a ^ 1.

It is easy to see that under given assumptions Wa is closed and equivalence of the norms holds. To finish proof of (i) assume that the norm || || is smooth and let w G E be given with ||w||a = 1. Suppose that x* G E* and

(3.10) (JC*, w) = 1, (x*9 w) ^ \\w\\a for all w G Wa. Then by (3.8) there exist 0 ^ X ^ 1,-1 ^ £ ^ 1 and b\ b2 G B such that (3.11) w = Xb] + (1 - \)(J3y + ab2).

Since ||w||a = 1 only the following cases are possible:

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 439

(a) X = 1 and l^'ll = 1, (b) A = 0 and ||62|| = 1, (c) 0 < A < 1 and \\b]\\ = ||Z>2|| = 1. — 1 — 9 Let b denote b if (a) holds and let b denote b if (b) or (c) hold. Then it follows from (3.10) that if y* := (x*, b)~ x* then

(y*9 b) S (y*, b) for all b e B and (y*, b) = 1. However, since the norm 11 11 is smooth there is the unique functional in E* which satisfies the above conditions. This in turn implies that the conditions in (3.10) are satisfied by the unique functional x*, hence the norm || ||a is smooth. Assume now that the norm || || is Kadec. Let a sequence wn, n

(3.12) ||wj|a =1 for n e N and wn —> w0 weakly. Then by (3.8)

2 (3.13) wn = \nb\ + (1 - \n)(/3ny + ab n) l for some \„, bn, b\, b\ such that 0 ^ \n ^ 1, -1 ^ Pn ^ 1, b n, b\ e B, n e N. Note that if a ^ 1, we may assume by (3.9) \n = 0, b\ = 0 for all n e N. The weak (sequential) compactness of the ball B if E is reflexive or the compactness of the set [ —1, \]y when a ^ 1 imply, that there exist X0 l and /?0, limit points of the sequences Xn and fin and there exist b 0, b0, weak 2 limit points of the sequences b\, b n. Hence

(3.14) w0 = AX + (1 - WoJ + «*o). and without loss of generality we assume that all considered sequences converge instead of taking their subsequences. Again only the three cases are possible. If X0 = 0, then for n sufficiently large 0 ^ \n < 1, which together with ||w0||a = I Wla = 1 implies

llftoll = II^H = 1- 2 Since B is Kadec we obtain b n —> b\. Hence also wn —» w0, because A„^0. 1 1 If X0 = 1, the analogous argument shows that bn —» b0, hence also wn —> w0, because (1 — Xn) —» 0. If 0 < X0 < 1, then we may prove that 2 K\\ = \\bi\\ = \\bi\\ = \\b n\\ = i

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 440 J. M. BORWEIN AND H. M. STROJWAS

for n G N sufficiently large and

h\ -* bl bn "^ b0

on using Kadec property of the ball B. Hence also wn^w and the ball Wa is Kadec.

To prove (ii), suppose that sequences wn e E, x* e E*, n e N\{0} and w0 e E9 XQ e £* are given such that

(3.15) wn -> w0, ||w0||a = ||wj|a = 1,

(3.16) (JC*, wn) = 1, (**, w) ^ 1 for all ^6^,«EN.

To prove that the norm || ||a is Frechet differentiable it is enough to show that (3.17) x* —> XQ in norm, while (i) implies that (3.18) x* —> x$ weak star.

(See for example [8] for justification.) We may assume that wn and w0 are 2 as in (3.13) and (3.14). Let bn9 b0 denote b n, b\, respectively if A0 = 0 and biv b0 if 0 < X0 ^ 1. Then whenever n is sufficiently large, (3.15) when a ^ 1 and (3.15) together with weak sequential compactness and Kadec property of the ball B when a > 0 imply

(3.19) ||F0|| = \\bn\\ = 1 and5w->60, while (3.16) implies

(x*, bn) ^ (x*, fc) for all Jefi, (where again without loss of generality we use the sequences instead of the subsequences). Define

Then

(3.20) (*£, fcw) = 1, (x^, ft) ^ 1 for all6eB,«e N. Now (3.19) and (3.20) together with the assumption that the norm || || is Frechet differentiable imply that xf —> xr in norm. Using this and (3.18) we get (3.17) which finishes the proof.

LEMMA 2. Let E be a smooth Banach space and let B, JVa, || ||, || \\abe as in Lemma 1. Suppose C is a closed subset of E and 0 £ C. Let w e C be such that

(3.21) \\w\\a = inf{ Hzlljz e C},

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 441

and denote the derivative of the norm \\\\a at w by x*. Then there exists y e E,y £ C such that (3.22) ||* - y|| = inf{ \\z - y\\ \z G C) and x£-, the derivative of the norm || || at w — y, is some positive multiple of x*.

Proof Assume without loss of generality that ||w||a = 1 and suppose that w is given by (3.11) and b is defined as in the proof of (i) in Lemma 1. Let y : = 0 if X = 1 and let y := Xb] + (1 - X)Py otherwise. Then if À = 1, we have for any z e C

11* - y\\ = Hfc'H = 1 = ||w||a =i ||z||a ^ IUH = \\z - JH, hence (3.22) holds. Also similarly as in Lemma 1 we get x*_y = xf = (x*, b)~]x*. So assume now that 0 ^ X < 1 and suppose to the contrary that \\c ~ y\\ < \\w - y\\ = (1 - X)a for some ~c in C. Then there exists 0 > 0, such that (3.23) c e y + ||c - y||£ c XZ)1 -f (1 - X)/3y + (1 - X - 0)aB. Assume 0 ë |0| < 1, then for 0 < (1 - X)(l - |j8| ), (3.23) shows c e (i - 0)^

which contradicts (3.21) as |H|a = 1. Similar contradiction follows also when \f}\ = 1. This proves (3.22). Furthermore X$_y = X$_y = Xf = (X*, b)~lX*, and the proof is finished. Let E be a Banach space and let A be a multifunction ,4:£h-*2£ Then the following inclusions are straightforward, (3.24) d lim inf A (*') c (norm)lim inf A (x') x'^x x'^x c (w* lim supyl0^))0,

where

w*(sequential)lim sup V4°(JC') x'—>x

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 442 J. M. BORWEIN AND H. M. STROJWAS

is the set of weak star limits of bounded sequences y* with y* e A°(xn) and xn —> x, while d lim inf A (V)

(the discrete limit inferior) is defined as follows (3.25) d lim inf A (*') : = U C\ A (x'\

where B is the unit ball in E. Recall that (norm)lim inf A(x') := n U n 4(x') + e£. 7 v y v y^x €>0 5>0 x'^x + 8B

THEOREM 3.1. If C is a closed subset of a reflexive Banach space E with Kadec and Frechet differentiable norm, x e C, then

(3.26) rc(jc) = lim inf WKc(x') x'—>x C

= lim inf WPc(x') = lim inf Bc(x') x'—*x x'—*x C C and the Frechet normal formula and the proximal normal formula

(3.27) Nc(x) = œw* lim sup WK°c(x'\ x'—*x C

Nc(x) = côw* lim sup PNc(x') y—»* C hold Proof. Using Theorem 1.2, Proposition 1.1, Proposition 3.2 and the last inclusion in (3.24) we get

(3.28) Tc(x) c lim inf WKc(x') c lim inf WPc(x') x'^x x'-^x C C

c lim inf Bc(x') c (w* lim sup PNC(JC') )°. x'—*x x'-^x C C From (3.24) and Corollary 3.1 it follows that

(3.29) lim inf WKc(x') c (w* lim sup WK°c(x') )° xf—*x x'—>x C C

c (w* lim sup PNC(JC'))°. We prove that

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 443

(3.30) (w* lim sup PNC(JC') )° c Tc(x). x'—>x C Let B be the closed unit ball in E and let || || denote the norm of £, which is by our assumption Kadec and Frechet differentiable.

Suppose^ £ Tc(x) and ||^|| = 1. Using the characterization of Tc(x) given in [18] or Penot [15], [16], there exists e > 0 such that for each n e N we can find \n > 0 and xn e C n (x + 2~"£) with

(3.31) C n(xn + (0, XJ(j> + 2eB) ) = 0.

Without loss of generality we suppose € < 1/4. Let us choose pn > 0 with _ pn < min(2 ", X„/4, c/4)

and an > 0 such that

aw < min(€pn/6, X„/6). Let us set for n e N

»;:= co(€5, (1 + any\y + a^), (1 + aJ-V-jy + a„€5))

= co((l + any\[-\,X\y + aneB),€B).

Then for all n e N ei? c Wn c 5. By Lemma 1 the norm || ||w associated with the ball Wn is equivalent with the norm || || and it is Kadec and smooth. Select, using Lau's nearest point result (Proposition 3.4), some yn in E with

(3.32) yn - y e anWn and

\\yn -*A = inf{ll^ ~z\\n\z e cn}9

for some xn e C„ := (C — x„) n p„5. We have y e (1 + an)Wn, hence

||.y„ - x„||n ^ lb„ - 0||„ ^ ||j„ - y\\n + \\y\\„ =i 1 + 2«„, so that

(3.33) xn = xn - yn + yn - y + y G (1 + 2«J»; + a„P^ + y = y + ^ + 3<*„tf;. Now 1 (3.34) y + Ww c co(j + €5, (1 + aj" (2 + «n)(y + c*), l (1 + any an(y + c5) ) c (0, + oo)(j; + e5) and

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 444 J. M. BORWEIN AND H. M. STROJWAS

((0, +OOXJ> + eB) + 3anWn) n C„

= { ((0, +OOXJ> + *B) + 3a„W„) n p„B) n C„ c { (0, (1 - €)-'(p„ + 3a„)](>> + £5) + 3a„W;} n C„

<= {(0,\n](y + tB) + 3anW„) n C„ c {(0, 3a„€-')(7 + ^) + 3a„H/} n C„

U {3a„£^', \n](y + (B) + 3«„W;} n C„ c {(0, 3«„€"1)(^ + «5) + 3a„^„} n C„ u {[Sa^-'AK.y+ 2ci?)} n c„

= { (0, 3a„€-')(j + eB) + 3

x„ e (0, 3ane~')(y + eB) + 3%Wn. Therefore

\\x„\\ < 3anr\l + e) + 3an = 3«„e"'(l + 2c) = 6"««~' < P„.

and we conclude that there exists rn > 0 such that

(3.35) \\xn - >g|„ = inf{ \\z - .yjjz e (C - *„) n (*„ + r„5) }.

Let >>* e £* be the derivative of the norm || ||n at yn — x„. Using (3.35), by Lemma 2, we choose yn e E such that

(3.36) ||x,; - yn\\ = inf{ \\z - yn\\ \z e (C - x„) n (x„ + /•„*) }

and A,? > 0 such that

(3.37) y*n = \yl--Xn,

where yf _- is the derivative of the norm || || at J77 — 3cw. Now it is not difficult to see that (3.36) and (3.37) imply that^*, n e N is the proximal normal functional to Catxw + 3cw as defined in Definition 3.1, that is

(3.38) y* G PNc(x„ + xw), /i G N. As E is reflexive the closed unit ball of E* is sequentially weak* compact, hence y* being bounded contains a weak star convergent subsequence. Without loss of generality assume that y* converges to some y* e E*. Then as xn -f xn —> x, we have by (3.38)

(3.39) y* e w* lim sup PNc(x')- C

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 445

But as

(y + W„) n Cn c (0, +00X.K + €B) n C„

c (0, Xn)(y + eB) n C„ = 0, as shown above, we have

Thus (y%y) = (J*^„ - *„) + Oft.y - ^ + *„)

= !-«„-«„- Pne~l > 1/2> which shows that (y*, y) > 0. This by (3.39) gives

y £ (w* lim sup PNc(x') )°, x'^x C which finishes the proof of (3.30). (3.28), (3.29), and (3.30) imply (3.26). Taking the polars of the sets in (3.28), (3.29), (3.30) and using the fact that if E is reflexive then the weak star closed convex sets in E* are norm closed, we get (3.27) and the proof is finished.

COROLLARY 3.2. If C is a closed subset of a reflexive Banach space, x e C then

(3.40) Tc(x) = lim inf WKc(x') = lim inf WPc(x') x'-*x x'^x C C and

(3.41) Nc(x) = œw* lim sup WK°c(x'\ x'—*x C Proof Combine Proposition 3.3 and Theorem 3.1. The first equality in (3.40) was proven in [14] under stronger assumptions on the set C Note that proximal normal formula (3.6) and the corresponding equalities for finite dimensional spaces obtained in [14], [7], [11], follow from Theorem 3.1. We complete Theorem 1.2 and Theorem 3.1 with the following important observation.

THEOREM 3.2. Let E be a Banach space. Then the following are equivalent.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 446 J. M. BORWEIN AND H. M. STROJWAS

1) lim inf WPc(x') = Tc(x) C for all closed sets C c E, x e C,

f 2) lim inf WPc(x ) D TC(X) x'—>x C for all closed sets C c E, x e C,

3) lim inf WKC(JC') D rc(x) x'^x C for all closed sets C c E, x e C, 4) E is reflexive. Proof Clearly 1) implies 2) and 3) implies 2). Theorem 1.2 shows that 3) is implied by 4) and Theorem 3.1 shows that 1) is implied by 4). Hence it suffices to show that 2) implies 4). Let E be irreflexive. Choose z e E and z* e £* with (3.42) (z*, z) = ||z|| = 1 = \\z*\\. Let y e £ be such that (3.43) (z*,j) = Oand \\y\\ = 1. Choose y* e £* with (3.44) (/»,;,) = 1 = 11^*11. Then

Ë := {x ^ E\ (z*, JC) = 0, (7*, JC) - 0} is obviously an irreflexive subspace of E. Let us observe that the unit sphere in E

S := {x <= £| ||JC|| = 1} is not countably relatively weakly compact. Indeed, otherwise by Eberlein's theorem (see for example [10] ) it is relatively weakly compact, which implies that cô S := [x ). W = I tt = 1

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 447

We will verify that

x (3.45) WPc(m~ z) = 0 for all m G N and

(3.46) y e 7C(0). As m~]z G cl C and since both the Clarke tangent cone and the weak contingent cones always are the same for a set and its closure, this will violate 2). x To prove (3.45) suppose h G WKc(m~ z), h ¥= 0. Then there exists M > 0 and a sequence hn with the weak limit h such that x tnhn + rn~ z ^ C

for some tn > 0, tn —> 0 and ||/zj| < M. Thus there also exist sequences ra„, £„, with mn, kn G N, ra„, kn > 0 and À„ with À„ ^ 1 for which _1 X 1 m z + tnhn = m~ z -f \nk~\m~ xK + j).

As (z*, xk) = 0 for « G N, this together with (3.42) and (3.43) imply 1 x m' - m~ = tn(x*, A„)->0.

Hence without loss of generality we assume mn = m for all « G N. Then = k l m 'A K n ( n\n + J0- As (/*, x^ ) = 0 for « G N this and (3.44) show x ] tn(y\hn) = \nK ^k; .

This in turn implies that kn —» oo. Let

Then

0A - ^ = ™~\n' Select /z* G E* with (/z*, /z) = ||/z||, \\h*\\ = 1. Then for n sufficiently large (h*,h„) ^ 2~1||/z||, and as

Pn(h*,hn) ^ \\h\\\\y + m-\j| ^ 2,

/?„ remains bounded. It follows now that ra~ xA has a weak cluster point. Since kn tends to oo, this means xn has a cluster point. This contradiction shows x x WPc{m~ z) = WKc(m~ z) = 0.

To prove (3.46), let sequences fw, zn be given such that, t„ -> 0, /„ > 0, z„ -> 0, z„ G C, then

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 448 J. M. BORWEIN AND H. M. STROJWAS

X 2 zn = mn z + \nkn \mn xK + y\

for some sequences Xm, mn, kn as above. As (z*, z) = 1, (z*,*^) = (z*,j>) = 0,

we may argue that mn —> 00. Setting _1 l 2 c„ := m„ z + (\nk~ + tn)(m~ xkn + j>), we have c G c ! « ^ C C (cw - zw)->.y and ^ G ^c(^)- (3-46) is proven and the proof is finished. If E is an infinite dimensional Banach space, then replacing the with the norm topology in the construction of the set C above, we recover the example from [18]. As a consequence we get the following.

THEOREM 3.3. Let E be a Banach space. Then the following are equivalent. f 1) lim inf Pc(x ) = Tc(x)

for all closed sets C c E, x e C,

2) lim inf i>c(jc0 D Tc(x) x'—*x C for all closed sets C c E, x G C,

3) lim inf Kc(x') D Tc(x) for all closed sets C cz E, x ^ C, 4) E is finite dimensional. The analysis of the proof of Theorem 3.1, Proposition 1.1 and Lemmas 1 and 2 shows that similar methods may be used to prove the following theorem.

THEOREM 3.4. If C is a weakly compact subset of a Banach space E, x e C then

lim inf Pc(x') c Tc(x), x'—>x C whenever E may be given an equivalent smooth norm, and

lim inf WPc(x') c Tc(x), x'—>x C whenever E may be given a Erechet differentiable norm.

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 449

4. "Discrete lim inf" inclusions and stars of boundedly relatively weak compact sets. It is an easy consequence of the corresponding definitions that if

lim inf Rc(x') c Tc(x) x'—*x C then

d lim inf Rc{x') + x' c lim inf Rc(x') 4- x' c rc(x) + x. x'—*x x'—*x C C Corollary 2.1 implies that if C is a closed subset of a Banach space, then

c n #r0O + *' star C,

whenever for all x E C

J lim inf Rc(x') + x' c Tc(x) + JC. x'—>x C This motivates our following considerations for the boundedly relatively weakly compact sets. First we recall Borwein's nearest point result proven in [3].

PROPOSITION 4.1 (Borwein). If C is a norm-closed relatively weakly compact subset of a Banach space E, which has a Kadec norm and is such that E* has an equivalent strictly convex , then the set of those points which have a nearest point in C is dense in E. THEOREM 4.1. Let E be a Banach space and let C be a norm-closed boundedly relatively weakly compact subset of E, x G C, then (i) the following inclusion holds

(4.1) d lim inf Pc(x') + x' c Tc(x) + x;

(ii) if E can be given an equivalent Frechet differentiable and Kadec norm then also

(4.2) d lim inf WPc(x') + x' c Tc(x) + x. x'—*x C Proof Let us observe first that as C is boundedly relatively weakly compact, then E : = span C is a weakly compactly generated subspace of E (see [2] for the proof) and as such it possesses an equivalent smooth and Kadec norm. ( [8] ). Let || || denote some equivalent norm on E which is smooth and Kadec in the case (i) and which is Frechet and Kadec in the case (ii). Let B be a unit ball associated with the norm || ||. Suppose that

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 450 J. M. BORWEIN AND H. M. STROJWAS

y £ Tc(x) + x and \\y - x\\ = 1.

Then as in the proof of Theorem 3.1 we can find e > 0, Xn > 0 and n xn G C n (x + 2~ B) for n e N with

(4.3) C H(xn + (0, \n](y - x + 2eB) ) = 0.

Without loss of generality suppose that e < 1/4 and choose pn > 0 with

Pn < min(2~", Xw/4, c/4)

and an > 0 such that

an < min(€pn/12, A„/6).

Let us set for « G N H/ := (1 + aJ-'U-l, lJO' - *„) + a„(€/2)B).

By Lemma 1, the norm || \\n associated with the ball Wn is equivalent norm on E, which is smooth and Kadec in case (i) and Frechet differentiable and Kadec in case (ii). Note also that there exists an equivalent strictly convex dual norm on E*, because E is weakly compactly generated ( [8] ). Select, using Borwein's nearest point result (Proposition 4.1), some yn in E with

X G a W and yn ~ (y - n) n n

11^ -xn\\n = inf{||^ - z\\n\z G Cn}9

for some xn e Cn := (C — xn) C\ pnB. Following the proof of Theorem 3.1 we get

\\yn - x„\\„ =S 1 + 2a„, Wn c 2B, and

(4.5) xn e y - xn + Wn + 3anWn. Also

(4.6) y - xn + Wn c (0, oo)(y - xn + (e/2)B)

= (0, + oo)(_y - x + (x - xn) + (e/2)B) c (0, oo)(j; - x + eB\ whenever 2~n ^ c/2. Let us restrict our considerations to those n for which this inequality is satisfied. Furthermore we get

{ (0, oo)(j - x + tB) + 3a„Wn} n C„

c { (0, \](y - x + cB) + 3a„Wn) n C„ ] = { (0, 6ane~ )(y - x + tB) + 3anWn) n C„,

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. PROXIMAL ANALYSIS 451

where the last equality follows from (4.3). Hence using (4.5) and (4.6) we have

||xj| ^ 6an€~\l + €) + 6an ^ Ma^ < pn,

which in turn implies that xn satisfies (3.35) for some rn > 0. Thus yn — xn is the proximal normal vector to C at xn + xn in the norm || ||M. However as

(y-x„+ Wn) ncn = 6 we get H** - yX = l ~ an- Thus

(4.7) \\yn -xn~(y -(xn + xn))\\n

^ an < 1 - an ^ \\xn - yn\\n. Now (4.7) implies in the case (i), via (1.3), that

y - (xn + xn) £ Pc(xn + xn)9 which proves (4.1) and in the case (ii), via (1.4), that

y-(xn + xn) £ WPc{xn + xn\ which proves (4.2).

COROLLARY 4.1. If C is a norm-closed boundedly relatively weakly compact subset of a Banach space E, then

(4.8) n Pr(x) + x = star C, and

n Pc(x) = rec(star C)

whenever C is starshaped. If E can be given an equivalent Frechet differentiable and Kadec norm then

(4.9) n WPc(x) + x = star C, and

n WPc(x) = rec(star C)

whenever C is starshaped. In particular (4.9) holds for closed subsets in reflexive spaces. Proof Combine Theorem 4.1 and Corollary 2.1. Note that always

Pi Pr(x) c rec( Pi PAx) + x) and

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use. 452 J. M. BORWEIN AND H. M. STROJWAS

n WPAx) c rec( n WPAx) + x),

which follows from the convexity of the corresponding cones.

This completes Part I. In Part II we continue on applications of this theory as indicated at the introduction.

REFERENCES

1. E. Bishop and R. R. Phelps, The support Junctionals of a , Proc. Symp. Amer. Math. Soc. 139 (1963), 433-467. 2. J. M. Borwein, Tangent cones, starshape and convexity, Internat. J. Math. & Math. Sci. 1 (1978), 459-477. 3. Weak local sup port ability and applications to approximation, Pacific Journal of Mathematics 82 (1979), 323-338. 4. J. M. Borwein and R. O'Brien, Tangent cones and convexity, Canadian Math. Bull. 19 (1976), 257-261. 5. J. M. Borwein and H. Strojwas, Tangential approximations, to appear in Nonlinear Anal, Theory, Meth. Appl. 6. F. H. Clarke, Optimization and nonsmooth analysis (John Wiley, 1982). 7. B. Cornet, Regular properties of tangent and normal cones, (to appear). 8. J. Diestel, Geometry of Banach spaces — selected topics (Springer-Verlag, 1975). 9. I. Ekeland, On the variational principle, J. Math. Anal. Appl. 47 (1974), 324-353. 10. R. B. Holmes, Geometric and its applications (Springer-Verlag, 1975). 11. A. D. Ioffe, Approximate subdifferentials and applications. 1: The finite dimensional theory, Trans, of the Amer. Math. Soc. 281 (1984), 389-416. 12. M. Krasnoselski, Sur un critère pour qu'un domaine soit étoileé, Rec. Math. [Math Sbornik] N.S. 19 (1946), 309-310. 13. K. Lau, Almost Chebyshev subsets in reflexive Banach spaces, Indianna Univ. Math. J. 27 (1978), 791-795. 14. J. P. Penot, A characterization of tangential regularity, Nonlinear Anal., Theory, Meth. Appl. 5 (1981), 625-663. 15. A characterization of Clarke's strict tangent cone via nonlinear semigroup, (to appear). 16. The Clarke's tangent cone and limits of tangent cones, (to appear). 17. R. T. Rockafellar, Proximal subgradients, marginal values and augmented Lagrangians in nonconvex optimization, Math, of Operations Research 6 (1981), 424-436. 18. J. S. Treiman, A new characterization of Clarke's tangent cone and its applications to subgradient analysis and optimization, Ph.D. Thesis, Univ. of Washington, (1983).

Dalhousie University, Halifax, Nova Scotia; Carnegie-Mellon University, Pittsburgh, Pennsylvania

Downloaded from https://www.cambridge.org/core. 29 Sep 2021 at 23:55:26, subject to the Cambridge Core terms of use.