MATH. SCAND. 102 (2008), 73–100 WEAK LINEAR CONVEXITY AND A RELATED NOTION OF CONCAVITY LARS HÖRMANDER 1. Introduction Linear convexity is a property of open sets in Cn which is stronger than pseudo- convexity but weaker than convexity: Definition 1.1. An open set ⊂ Cn, n>1, is called linearly convex if every ζ ∈ ֙ is contained in an affine complex hyperplane ζ ⊂ ֙, and is called weakly linearly convex if this is true when ζ ∈ ∂. The notion was first studied by Behnke and Peschl [2] when n = 2, long before the geometric conditions for pseudoconvexity were fully understood. A renewed interest has been created by the study of analytic functionals. Weak linear convexity implies pseudoconvexity. If R(z) denotes say the euclidean distance from z ∈ to ∂ then is pseudoconvex if and only if − log R is plurisubharmonic in , or equivalently, n n 2 ¯ 2 n (1.1) tj tk∂ R(z)/∂zj ∂z¯k ≤ tj ∂R(z)/∂zj /R(z) in , t ∈ C , j,k=1 1 in the sense of distribution theory. Note that R is Lipschitz continuous with Lipschitz constant 1. By Rademacher’s theorem ∂R/∂zj is defined almost everywhere and bounded, so the right-hand side of (1.1) is a well defined func- ∞ ∈ 2 ∈ 2 tion in Lloc().If∂ C , then R C in a neighborhood of ∂ if defined ¯ 2 with a negative sign outside , and (1.1) implies that tj tk∂ R(z)/∂zj ∂z¯k ≤ 0 ∈ = 2 when z ∂ and Rz(z), t 0. If is a C defining function of , thus <0in, = 0 and d = 0on∂, this means that (1.2) n n ¯ 2 n tj tk∂ (z)/∂zj ∂z¯k ≥ 0ifz ∈ ∂, t ∈ C , tj ∂(z)/∂zj = 0. j,k=1 1 Received August 24, 2006. 74 lars hörmander Conversely (1.2) implies (1.1) and thus pseudoconvexity. The primary aim of this paper is to study similar conditions for weak lin- ear convexity. Let us first recall two well-known basic results. (See e.g. [3, Proposition 4.6.4 and Corollary 4.6.5].) Proposition 1.2. Let ⊂ Cn be a bounded connected open set with C1 boundary, and assume that is locally weakly linearly convex in the sense that every ζ ∈ ∂ has a neighborhood ω such that ω ∩ TC(ζ ) ∩ =∅; here TC(ζ ) is the complex tangent plane of at ζ, that is, the affine complex hyperplane through ζ contained in the tangent plane. Then is weakly linearly convex. Proposition 1.3. Let ⊂ Cn be a (locally) weakly linearly convex open set with a C2 boundary, and choose a defining function ∈ C2(Cn). Then it follows that the second differential d2 of is a positive semidefinite quadratic form in the complex tangent plane TC(ζ ) at ζ ∈ ∂. Conversely, if is open, 2 bounded and connected, and d is positive definite in TC(ζ ) for every ζ ∈ ∂, then is weakly linearly convex. Proposition 1.3 is in fact a very easy corollary of Proposition 1.2. – In two more recent papers [4], [5], Kiselman has proved that in the last statement it suffices to assume that d2 is positive semidefinite, which gives a charac- terization of bounded connected weakly linearly convex open sets with a C2 boundary in terms of a pointwise convexity condition on the boundary. How- ever, since the localization principle in Proposition 1.2 is valid when ∂is just in C1, it is natural to ask if Kiselman’s result can be extended to this case. The necessary “Behnke-Peschl condition” in Proposition 1.3 has a natural analogue (1.3) lim (w)/|w − ζ|2 ≥ 0, TC(ζ ) w→ζ where ζ ∈ ∂ and is a C1 defining function for . Since the ratio between two C1 defining functions is bounded, this condition is independent of the choice of defining function. If ∈ C2 then (1.3) means precisely that d2 is positive semidefinite in TC(ζ ), so (1.3) is an extension of the usual Behnke- Peschl condition to the case where ∂ ∈ C1. The necessity of (1.3) is obvious when ∂ ∈ C1 but we have only been able to prove the sufficiency under stronger regularity conditions. Before stating the result we shall recall some simple facts concerning open sets ⊂ RN with C1 boundary. (It is convenient to use real notation for a moment.) As above let R(x) be the distance from x ∈ to ∂, and let h = R2, thus (1.4) R(x) = inf |x − y|, h(x) = inf |x − y|2,x∈ . y∈∂ y∈∂ weak linear convexity and a related notion of concavity 75 The functions R and h are Gateau differentiable everywhere, h(x +˜x) = h(x) + h(x;˜x) + o(|˜x|), x˜ → 0; h(x;˜x) = sup{2˜x,x − y; y ∈ ∂,|x − y|2 = h(x)}. (See e.g. [3, Lemma 2.1.29].) Thus h is differentiable at x if and only if there is only one y ∈ ∂with minimum distance to x. Since R is Lipschitz continuous by the triangle inequality it follows from Rademacher’s theorem that R, hence 2 also h = R , is differentiable for almost every x ∈ .Ifδ ={x ∈ ; R(x) < 1 1 δ} and ∂ ∈ C , then R ∈ C (δ) if and only if every x ∈ δ is contained in an open ball ⊂ with radius δ. If this is true for some δ>0 we shall say that satisfies the interior ball condition.Ifν(y) is the interior unit normal of ∂ at y ∈ ∂, then ∂×(0,δ) (y, t) → y +tν(y) ∈ δ is a homeomorphism. We can now state an improvement of the results of Kiselman [5]: Theorem 1.4. Assume that ⊂ Cn is open, bounded and connected, that ∂ ∈ C1 and that the interior ball condition is fulfilled. Let be a C1 defining function of .If (1.3) is valid for every ζ ∈ ∂, and for some constant C 2 (1.5) (w) ≥−C|w − ζ | ,ζ∈ ∂, w ∈ TC(ζ ), then is weakly linearly convex. The interior ball condition is fairly strong. In fact, ∂ is in C1,1 (that is, there is a defining function with Lipschitz continuous first derivatives) if and only if both the interior and the exterior ball condition are fulfilled. The exterior ball condition is stronger than (1.5) but closely related. The proof of Theorem 1.4 given at the end of Section 2 will be based on a study of the function h in defined by (1.4). As a preparation we rephrase the conclusion and the hypotheses of Theorem 1.4 in terms of h: Proposition 1.5. An open set ⊂ Cn is weakly linearly convex if and only if ≤ + − +| − |2 ∈ (1.6) h(w) h(z) 2 w z, hz(z) w z, hz(z) /h(z), w , when h is differentiable at z ∈ . We shall always use the notation ·, · for the bilinear scalar product in Cn and (·, ·) for the sesquilinear one. Proof. If is weakly linearly convex and h is differentiable at z ∈ , then =¯− ¯ | − |2 = hz(z) z ζ where ζ is the point in ∂ where z ζ h(z). The plane ζ of Definition 1.1 must be the plane through ζ with normal z − ζ for it does 76 lars hörmander not intersect the open ball with center z and ζ on the boundary. The right-hand side of (1.6) is equal to | − |2 + − ¯ − ¯+| − ¯ − ¯|2 | − |2 z ζ 2 w z, z ζ w z, z ζ / z ζ =|w − ζ|2 − |w − z|2 −|(w − z, z − ζ)|2/|z − ζ|2 . The parenthesis is the square of the distance from w to the complex line through z and ζ,sothe right-hand side of (1.6) is the square of the distance from w to ζ , which is ≥ h(w) since ζ ⊂ ֙. This proves (1.6); if h is not ¯ − ¯ differentiable at z then (1.6) remains valid with hz(z) replaced by z ζ for any ζ ∈ ∂ with |z − ζ |2 = h(z). Now assume instead that (1.6) is valid. Since h(w) > 0 when w ∈ it follows from the interpretation of the right-hand side just given that ζ = {w; (w−ζ,z−ζ) = 0} does not intersect . For an arbitrary ζ ∈ ∂and every ε>0 we can choose z ∈ with |z−ζ| <εso that h is differentiable at z.Ifζ˜ is the point in ∂ with |ζ˜ − z|2 = h(z), then |ζ˜ − ζ|≤|ζ˜ − z|+|z − ζ| < 2ε, → ֙ ⊃ ˜ and we have a plane ζ˜ ζ with ζ˜ . When ε 0 it follows that there is such a plane through ζ , which completes the proof. Using the first part of the proof we can easily convert the conditions (1.3) and (1.5) to properties of h: Proposition 1.6. If ∂ ∈ C1 and h is differentiable at z ∈ , then + − − −| |2 | |2 ≤ (1.7) lim h(z w) h(z) 2 w, hz(z) w, hz(z) /h(z) / w 0 w→0 if (1.3) is valid at the point ζ ∈ ∂with h(z) =|z − ζ|2.If (1.5) is valid, then + ≤ + + | |2 + ∈ (1.8) h(z w) h(z) 2 w, hz(z) C w ,z w , for some constant C independent of z and w when z and w are bounded. Proof. By the interpretation of the right-hand side of (1.6) given in the proof of Proposition 1.5, the meaning of (1.7) is that lim (h(z + w) −|z + w − ζ − w|2)/|w|2 ≤ 0, w→0 where ζ + w is the orthogonal projection of z + w on TC(ζ ), thus |w |≤|w|. If ζ + w ∈/ then h(z + w) −|z + w − ζ − w|2 ≤ 0 so there is nothing to prove.