METRIC NORMAL AND DISTANCE IN THE HEISENBERG GROUP

NICOLA ARCOZZI, FAUSTO FERRARI

Abstract. We study the smoothness of the distance function from a set in the Heisenberg group, with some applications to the mean curvature of a surface.

Contents 1. Introduction 1 2. Notation and preliminaries 3 3. The metric normal and the distance function for a plane 5 4. The metric normal and the oriented metric normal for a smooth surface 7 5. Regularity of the distance function near a smooth surface 11 6. The Hessian of the distance function for a smooth surface 17 7. Mean Curvature 22 8. Examples 25 8.1. Nonvertical planes 25 8.2. The Carnot-Charatheodory sphere and the Hessian of the distance function 27 References 29

1. Introduction In this article we investigate some properties of the function “distance from a surface” in the Heisenberg group. Let H = H1 be R3 with the Heisenberg group structure and let d be the associated Carnot-Charatheodory distance (see Section 2 for the denitions). If S H is closed, the distance from a point P to S is

dE(P ) = inf d(P, Q) Q E ∈ Here, we consider the case where S = ∂ is the boundary of a C2, open subset of H, in the Euclidean sense. The choice of considering C2 surfaces in the Euclidean sense is motivated by the fact that such surfaces satisfy a internal/external Heisenberg sphere condition at any non- characteristic point, as we shall see below. Most results in this paper can be extended to open C2 surfaces in H. It is convenient to introduce the signed distance from S,

dS(P ) if P (1) S(P ) = ∈ d (P ) if P / ( S ∈ Observe that, given S, the signed distance S is dened modulo a sign, which corresponds to the choice of one of the two open sets having boundary S, or, equivalently, to the choice of an orientation for S.

Date: 14/10/2003. MSC 49Q15, 53C17, 53C22 1

2 NICOLA ARCOZZI, FAUSTO FERRARI

Our motivation for studying distance functions is twofold. In a recent article [12] and [13] it was shown, in particular, that the function dE is a viscosity solution of the eikonal equation in H, d = 1 |∇H E| (here, is the horizontal ) hence of the innite Laplacian equation, see [5] and [10] ∇H 2 , u = XiuXju(XiXju). ∞ H i,jX=1 It is interesting to have more infromation on particular solutions of these equations. Another reason for investigating dS is that a useful tool for the study of smooth surfaces S in Rn is the fact that the Euclidean distance from S has maximum growth, at least near S, along the segment normal to S. We introduce a denition of metric normal to a smooth surface which plays precisely this role in the Heisenberg group. The metric normal to S at P , P a point of S, is the set P S containing all points Q such that dS(Q) = d(P, Q). It will be shown that is an arc of aNgeodesic and an explicit formula for S in terms of the parametrization of SNwill NP be given. The available denition of horizontal vector normal to a surface in H is related to the metric normal, but it does not exhaust the concept. In fact, there exist dierent surfaces passing through a point P , having the same horizontal normal vector, but dierent metric normals, their common feature being that their velocity at P is a multiple of the the horizontal normal vector. Roughly speaking, the possible horizontal normal vectors are in a 1-to-1 correspondence with the group of rotations in H, and for each choice of the horizontal normal vector, the possible metric normals are in a 1-to-1 correspondence with the family of dilations in H. The knowledge of the metric normal allows one, as in Rn, to prove the smoothness of the distance function.

Theorem 1.1. Let S = ∂ be boundary of a Ck open subset of H. Then, for any non- k 1 characteristic point P S there is a neighborhood of P in H such that S is C in . ∈ U U Moreover, if the surface S is the zero set of a smooth function g : H R, then, on the surface S, → g (2) = ∇H . ∇H S g |∇H | A point C of S is characteristic for S if the Euclidean space to S at C coincides with the horizontal ber at C. See Section 2 for precise denitions. The choice of + or depends on the choice of between the two open sets in H having S as boundary. Here, Ck is in the Euclidean sense. In Section 6 we compute an explicit expression for the horizontal Hessian (see Section 6 for

the denition) HessH dS of dS in terms of g. As a corollary, we obtain a new characterization of the mean curvature of a surface in S. Recall that, if S is a smooth surface in H dened by g(x, y, t) = 0 and P is a non-characteristic point of S, then the mean curvature hS(P ) of S {at P [9] is dened} as in Euclidean space [8], Xg(P ) Y g(P ) (3) h (P ) = X + Y . S g(P ) g(P ) |∇H | |∇H | Below, we assume that g(x, y, t) = t f(x, y), where f is a C2 function. Theorem 1.2. Let S = ∂ be boundary of a C2 open subset of H. Let Q = XXf (Y f)2 (XY f + Y Xf) Xf Y f + Y Y f (Xf)2.

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP1 3

Then, (Y f)2 Xf Y f Xf Y f f 5 Q 4 f 3 Y f f 5 Q 4 f 3 (4) Hess d = |∇H | |∇H | |∇H | |∇H | . H S  Y f Xf h(Xf)2 Xf Y f i Xf f 5 Q + 4 f 3 f 5 Q + 4 f 3 |∇H | |∇H | |∇H | |∇H | In particular, for any non-char acteristich P in S, i  (5) d (P ) = h (P ). H S S This note is structured as follows. In Section 2 we give notation and some preliminaries. The Section 3 contains some elementary properties of the metric normal and a description of the metric normal to S when S is a Euclidean plane in H. In Section 4 we prove that the metric normal to S at a non-characteristic point is a non-trivial arc of a geodesic, and we deduce in Section 5 that d is smooth in a neighborhood of S characteristic points of S . In Section 6 S { } we compute the horizontal Hessian of dS on S. In Section 7 we show that several denitions of mean curvature for a surface dS in H, some of them involving dS, are equivalent. Some examples are given in Section 8. We consider the case when = t = 0 is a plane, and we compute { } HessH d in H, not just on . Then, we compute HessH d, where = P : d(P, O) = 1 is the unit sphere at the origin, with respect to the Carnot-Charatheodory distance.{ This amoun} ts, in fact, to compute the horizontal Hessian of dO, the distance function itself. In this paper we considered the Heisenberg group with the lowest dimension. This object is interesting in itself. For instance, it is related with the study of the isoperimetric inequality in the Euclidean plane [11]. More recently, it was realized that sub-Riemannian structures modelled on the lowest dimensional Heisenberg group can be used to model the human visual system, see [3], [4], [17] and the references quoted therein. In particular, [4] contains some material related to curvature in Heisenberg-like structures.

2. Notation and preliminaries In this section, we collect some basic denitions and known facts about the structure and the geometry of H. There is a vast literature on sub-Riemannian geometry and Carnot groups. Justs to quote a few titles, we refer the reader to [2], [6], [7], [11], [14], [15], [16], [18]. The Heisenberg group H = H1 is the Euclidean space R3 endowed with the noncommutative product (x, y, t) (x0, y0, t0) = (x + x0, y + y0, t + t0 + 2(x0y xy0)). Sometimes it is convenient to think of the elements of H as (z, t) C R. The Carnot- ∈ Charatheodory distance in H is the sub-Riemannian metric that makes pointwise orthonormal the left invariant vector elds X and Y , X = ∂ + 2y∂ , Y = ∂ 2x∂ . x t y t The vector elds X, Y do not commute, [X, Y ] = 4∂ . The distance between two points P t and Q in H is denoted by d(P, Q). The span of the vector elds X and Y is called horizontal distribution, and it is denoted by . The ber of at a point P of H is P = span XP , YP . The inner product in is denotedH by , . H H { } HP h i An important element of H’s structure is the dilation group at the origin : = 0 , { 6 } 2 (z, t) = (z, t), z = x + iy By left translation, a dilation group is dened at each point P of H. The Heisenberg group is also endowed with a rotation group, which is useful in simplifying some calculations. For R, let ∈ i R(z, t) = (e , t) be the rotation by around the t-axis. Composing with left translation, one could dene rotations around any vertical line (x, y) = (a, b). R is an isometry of H and its dierential

4 NICOLA ARCOZZI, FAUSTO FERRARI

acts on the ber as a rotation by . Under the usual identication between the Riemannian HO tangent space of H at O, TOH, and the Lie algebra h of H, the dierential of R can be thought of as a rotation on span X, Y , the rst stratum of h. With respect to the basis X, Y , { } { } cos() sin() dR V = V sin() cos() whenever V span X, Y . With some abuse of notation, we denote dR by R . ∈ { } The distance between two points P and Q in H is dened as follows. Consider an absolutely 3 continuous curve in R , joining P and Q, which is horizontal. That is, ˙ (t) = a(t)X(t) + b(t)Y lies in . Its Carnot-Charatheodory length is l (), (t) H(t) H 2 2 1/2 lH() = (a(t) + b(t) ) dt Z The Carnot-Charatheodory distance between P and Q, d(P, Q), is the inmum of the Carnot- Charatheodory lengths of such curves. The inmum is actually a minimum, the distance between P and Q is realized by the length of a geodesic. By translation invariance, all geodesics are left translations of geodesics passing through the origin. The unit-speed geodesics at the origin [12], [11] are 1 cos() sin() x() = sin((W )) + cos((W )) sin() 1 cos() (6) () = y() = sin((W )) cos((W )) O,,W   sin() t() = 2 2 Here, W is a unitary vector in and (W ) [0, 2) is unique with the property ˙ (0) =  HO ∈ O,,W W . R, and the geodesic is length minimizing over any interval of length 2/ . In the case = 0,∈the geodesic is a straight line in the plane t = 0 , | | { } x() = cos((W )), y() = sin((W )), and we say that the geodesic is straight. If P = (z, t) and z = 0, then there exists a unique length minimizing geodesic connecting P 6 and O. If P = (0, t), t = 0, (i.e., if P belongs to the center of H) then there is a one parameter family of length minimizing6 geodesics joining P and O, obtained by rotation of a single geodesic around the t-axis. We call curvature of the parameter ( ) = . We will see later how ( ) O,,W O,,W | | O,,W is related to the curvature of a surface in H. The notion of curvature extends to all geodesics by left translations, (LP ) = (), where LP is left translation by P H in H. Given points ∈ P = (z, t) and P 0 = (z0, t0), they are joined by a unique length minimizing geodesic, unless z = z0. Let P,, = LP O,, The parameter is geometric in the following sense. 2/ is the length of and sgn() | | P,,W is positive if and only if the t-coordinate increases with . Recall that in H the orientation of

the t-axis is an intrinsic notion, unlike the Euclidean space. If P,,W and P 0,0,W 0 have an arc in common, then = 0, while no such easy relation exists for the parameter W . To change the orientation of a geodesic, observe that

P,,W () = P, , W ( ) Unlike the Euclidean case, a geodesic leaving O is not determined by its tangent vector at the origin, ˙ (0) = W = cos((W ))XO + sin((W ))YO. The extra parameter we need is (). The parameter (), if = () = 0, is related to the dilation group as follows. ( ) 6 O,,W is a reparametrization of the geodesic O,/,W . That is, all geodesics leaving 0 and having xed initial velocity ˙ (0) = v in are dilated of each other. The case of the straight geodesics HO

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP2 5

is the limiting one, corresponding to 0. In a precise sense, then, the set of geodesics at → O is parametrized by the unit circle in O and by the dilation group, a feature of H with no Euclidean counterpart. This simple factHwill be a recurrent theme in the following sections. Let S be a smooth surface in H. We need some dierential geometric notions about S. Denition 2.1. Let S be a smooth surface in H and let P S be a non-characteristic point. ∈ The Euclidean tangent space to S at P is denoted by TP S. The direction tangent to S at P is the 1-dimensional space V S = T S . The plane tangent to S at P , S, is the P P ∩ HP P Euclidean plane in H, tangent to S at P in the Euclidean sense. The direction normal to S at P is N S = V S. P HP P When P S is not a vertical plane in H, we dene a positive direction normal to S at + P , denoted by NP S, as follows. Let C be the characteristic point of P S. NP S is the direction normal, with respect to , , , to that of the straight line CP and N +S is the unit vector in h iH P the direction NP S which makes a (counterclockwise) angle /2 with the haline CP , having origin in C. The group tangent to S at P [15] is the 2-dimensional vector space G S = S , P VP T where = (0, 0, t) : t R is the center of H and P S is the one parameter subgroup of H T { ∈ } V generated by VP S. One point we want to make in the present paper is that GP S does not seem to capture the complexity of the geodesics’ set, while TP S does, in a precise sense. The following facts are easily established by direct calculation. Proposition 2.1. Let S be a smooth surface in H, implicitely dened by g(x, y, t) = 0. Let P S be non-characteristic. Then, ∈ (7) V S = span Y g X Xg Y , N S = span Xg X + Y g Y = span g P { } P { } {∇H } 3. The metric normal and the distance function for a plane In this section, we list the main results about the distance function for a plane. Denition 3.1. Let E be a subset of H, P E. The metric normal to E at P is the set ∈ P S of the points Q H such that d(Q, E) = d(Q, P ). N ∈ See also [1], where a pathological occurrence of the metric normal to the unit sphere was used to study bi-Lipschitz functions in H. Proposition 3.1. Let be a plane in H and let P be non-characteristic. If has a characteristic point C, then,P ∈ P P P = ([ d(P, C), d(P, C)]) P,2/d(P,C),N+ N P P P 2 2 If is a vertical plane, then is the straight geodesic through P , in the direction N . P NP P P P Another way to state this result is the following. The projection onto the t = 0 plane of is the circle c having as diameter the line joining C1, the projection of C, and P1, the projection of P . Proof. of the proposition. The case when is a vertical plane is elementary. Consider the case when has a characteristic point. P In theP statement of the proposition, everything is invariant under isometries, we can then left-translate everything so that is the plane t = 0 and C = O. Let P = 0 be a point in . Consider a point (0, 0, T ) on the vPertical axis so that a legth-minimizing geo6 desic from (0, 0, TP) to (0, 0, T ), say , intersects at P . = P,,W is uniquely determined. We need to show that the parameters of are thosePin the proposition’s statement, and that is the metric normal. Step 1: has the right parameter . Since points upward, 0. By symmetry with respect to t = 0, d(P, (0, 0, T )) = /, half-length of . Inserting this inthe equation for , and

6 NICOLA ARCOZZI, FAUSTO FERRARI

assuming after a rotation that P lies on the positive x-axis, we obtain that P = (2/, 0, 0). This shows that d(P, 0) = 2/. Step 2. lies in the metric normal. Let = (0, 0, T ). Consider the ball B = B( , d(P, )). The points Q where B touches a planeS above , parallel to , are those for S S S P which the geodesic between and Q has parameter 1 = /d(P, ) (i.e., the points having t- coordinates which is greatest).S But we just saw that has parameterS = 2/d(P, 0) = /d(P, ), hence P is one of such points. It follows that d(P, ) minimizes the distance between andS , hence that lies on the metric normal. S S P Step 3. has the right parameter W . Let c = c(s) be the projection of on , assuming that starts from T and goes towards T passing through P . Then, c(0) = c(2P/) = 0, c(/) = P and c isa circle having diameter OP . As s increases, c(s) runs the circle clockwise. As to the tangent vectors, c˙(s) is the projection on the plane = t = 0 of the horizontal vector ˙ (s). Suppose P = (2/, 0, 0) lies on the positive x-axis. PThen,{ c˙(/}) = (0, 1), hence + ˙ (/) = Y . We only have to show, then, that Y = NP . Now, the space tangent to at P containsjust one horizontal direction (P is non-c haracteristic),P and this direction is thatP of + + the x-axis, X. Then, NP = span Y , hence, since NP has to point upward, NP = Y , as wished. P { } P P Step 4. contains the metric normal. Suppose now that Q belongs to both and NP P P 0 . We show that, then, Q lies on the axis x = y = 0 and that P and P 0 lie on a Euclidean circleN P centered at O in . P After a group translation, the plane L 1 touches a closed Carnot Charatheodory ball B Q 1 1 P 1 1 centered at the origin in Q P and Q P 0. This may happen only if Q P and Q P 0 both lie on the circle c of the points in B having highest (or lowest) t-coordinate, and c = B L 1 . Q Hence, the characteristic point of lies on the same vertical line as the center of B,∩and thisP P property is invariant under left translations. As a consequence, Q lies on x = y = 0 and P , P 0 are taken one into the other by a rotation around x = y = 0. For each point P in O , consider the geodesic arc P { } P = ([ d(P, C), d(P, C)]) P,2/d(P,C),N+ P P 2 2 which, by steps 1-3 lies in P . As P varies, the union of the sets P lls up H O . N P { } P Suppose now that Q belongs to P P . Clearly, Q = O, and Q belongs to some 0 . By the considerations abovNe, Pthen, Q lies on x =6 y = 0 and P is taken to P by a NP 0 P 0 rotation around x = y = 0. The same rotation brings P 0 onto P and xes Q, hence Q P , a contradiction. ∈

By Proposition 3.1 and (6) we have an explicit expression for the metric normal to a plane. Theorem 3.1. The metric normal to = t = 0 at P = (z, t), z = x + iy, is the support of the geodesic arc { } x 2 y 2 2 1 + cos z + 2 sin z | | | | (8) () = (u(), v(), s()) =  y 1 + cos 2 x sin 2 , z P  2 z 2 z  2 | | | | | | 2 | |  z 2 2 | 2| z + sin z | | | |  Let w = u + iv. Then, by (8),  w = z cos z (9) | | | | | |  2 1 2 s = z z + 2 sin z  | | | | | | The next lemma helps with calculations.

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP3 7

Lemma 3.1. If p is the plane t = ax + by + c, its characteristic point is C = ( b/2, a/2, c) Proof. The tangent space of p at (x, y, t) is spanned by (1, 0, a) and (0, 1, b). They are both horizontal i a = 2y and b = 2x. 4. The metric normal and the oriented metric normal for a smooth surface Lemma 4.1. Let S be a smooth surface in H, implicitely dened by g(x, y, t) = 0. Let P = (x , y , t ) S. Then, if g (P ) = 0, S has a characteristic point C and 0 0 0 ∈ t 6 P g (P ) g (P ) g (P ) g (P ) (10) C = y , x , t + x y + x 2g (P ) 2g (P ) 0 0 g (P ) g (P ) t t t t g g (11) d(P, C) = 2 |∇H | = |∇H | [X, Y ]g 2 ∂ g | | | t | When gt(P ) = 0, we say that P S has characteristic point at innity. The quantity d(P, C) is purely non-Euclidean. The proposition below says that it is related to another purely non-Euclidean feature of H, that is, the role of the dilation group in the determination of geodesics. Theorem 4.1. Let be an open subset of H with C2 boundary S, in the Euclidean sense, locally dened by the equation g(x, y, t) = 0. Let P be a non characteristic point of S. Then the metric normal S to S at P is a proper subarc of the length-minimizing geodesic having N S NP P as tangent vector at P , and with () = 2/d(P, C). That is, P S = supp() = P supp() (left translation by P ), = (u, v, s), N d(P,C) Y g(P ) 2 Xg(P ) 2 u() = 2 g(P ) 1 cos d(P,C) + g(P ) sin d(P,C) |∇H | |∇H |  d(P,C) n Xg(P ) 2 Y g(P ) 2 o (12) () = v() = 2 g(P ) 1 cos d(P,C) + g(P ) sin d(P,C)  H H  |∇ | |∇ | s() = 1 d(P, Cn)2 2 sin 2 o 2 d(P,C) d(P,C) When ∂ g(P ) = 0, i.e. when the Eucliden an plane tangentoto S at P is vertical, formula (12) t  becomes Xg(P ) Y g(P ) (13) () = , , 0 g(P ) g(P ) |∇H | |∇H | If P is characteristic and S is smooth, then the metric normal at P contains P alone. First, we need the following lemma. Lemma 4.2. Let be an open subset of H with C2 boundary S, in the Euclidean sense. Then, for each non characteristic point P in S, there exist an open neighborhood of P in S and C > 0 such that, for any Q in there is a closed Heisenberg ball B in , containingU Q on its boundary. U Given lemma 4.2, we prove Theorem 4.1.

Proof. of the Theorem. Let P S the Euclidean plane tangent to S at the non characteristic point P , let B = B(R, ) a Heisenberg ball in , containing P on its boundary and be the length minizing geodesic between R and P ( is unique, otherwise S would not be smooth at P ). Choosing, if necessary, a point R0 on closer to P , and considering B(R0, d(R0, P )), we can assume that B(R , d(R , P )) B, by Lemma 4.4 below, and that B(R , d(R , P )) does 0 0 0 0 not intersect neither S nor P S and it contains P on its boundary. Then, the segment of the geodesic with endpoints P and R0 is contained in the metric normal to P S at P . By Theorem 3.1 and by the equation of P S, we obtain that the equation of is that given in (12).

8 NICOLA ARCOZZI, FAUSTO FERRARI

We have now to show that every other point Q of P S belongs to . Let Q be in P S and let be the length minimizing geodesic between Q andN P . If contains a subarc of Nstarting at P , then is contained in and there is nothing to prove. Suppose that, on the contrary, is a dierent geodesic. Then, possibly choosing a point Q0 nearer to P on , we can assume that is a portion of P P S. On the other hand, is portion of P P S, but in Proposition 3.1 we showed that theNmetric normal to a plane is contained in aNsingle geodesic, hence and are subarcs of the same geodesic. Finally, observe that S is a proper subarc of a length minimizing geodesic. Suppose, in NP fact, that P S is a complete geodesic starting at P and let P 0 be its other endpoint. By (6), P and P Nlie on the same vertical line. Then, R S is a geodesic whose length realizes the 0 NP distance between P and P 0, hence between P 0 and S. This means that R P S = P S, which is absurd. N N

Proof. of Lemma 4.2. Let P be a non characteristic point on S, let P S be the Euclidean plane tangent to S at P and let be the metric normal to S at P pointing inside . Let s 0 be P a number smaller than the length of and consider the point Cs on having distance s from P . Let s be the left translation carrying Cs in O and let S0, P 0 0, 0 be the images of S, P and , , respectively. By continuity and by the assumption that S is C2, there is r > 0 such s that, for each s < (r), there exists a Euclidean ball BEuc(Ds, r) of radius r contained in 0 and having P 0 on its boundary. We will show that there exists a Heisenberg ball B(R, ) centered in point R of 0, contained s 1 1 in BEuc(Ds, r) and having P 0 on its boundary. The ball s B(R, ) = B(s R, ) fullls the requirements of the lemma for Q = P . Since our procedure is stable with respect to P , it provides the required ball for Q in a neighborhood of P in S, with radius bounded from below by a positive constant. s Since P 0 is non characteristic, a subarc of 0 starting at P 0 is contained in BEuc(Ds, r), but s for the point P 0. It suces to prove the following claim: BEuc(Ds, r) contains a Heisenberg ball having P 0 on its boundary. We need an estimate for the Euclidean curvatures of the Heisenberg ball with center at the origin. Lemma 4.3. Let B(0, s) be the Heisenberg ball centered in 0 with radius s. For every point P ∂B(0, s), P = P (x, y, t), (x, y) = (0, 0), then there exists a positive number s0(P ) such that the∈principal Euclidean curvatures in6 P are respectively, as s 0: → E 1 k1 = (1 + o(1)) 1 2 s 1 + 16 q and 3 kE = (1 + o(1)). 2 4s3 where o(1) 0 as s 0, uniformly w.r.t. C, for any xed C > 0. → → | | s Proof. ¿From (8) and = 2 we get the following parametrization of the Carnot ball of radius s: sin x = s cos sin y = s sin  s2 2 sin(2)  t = 2 2 . E E We can calculate the Euclidean curvatures k1 , k2 of the Carnot ball obtaining respectively g () kE = 0 1 2 s 2 f() f 0() + ( 2 g0()) p

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP4 9

and E 1 g0() k = ( )0, 2 f () 2 s 2 0 f 0() + ( 2 g0()) sin 2 2 sin(2) where f() = s and g() = s 22 p. As a consequence we get G () kE = 0 1 2 2 F () F 0() + (sG0()) and p E 1 G0() k = ( )0, 2 2 2 F 0() F 0() + (sG0()) sin 2 sin(2) where F () = and G() = 2 .pOn the other hand as 0 for xed , 2 → 1 3 4 F 0() = + + o( ), 3 30 2 2 2 3 G0() = + o( ), 3 5 and 1 1 2 3 F 00() = + + o( ) 3 10 4 2 G00() = + o( ). 5 Hence E 1 k1 = (1 + o(1)) 1 2 s 1 + 16 and q 3 kE = (1 + o(1)). 2 4s3

We also need this fact. Lemma 4.4. Let B = B(A, r) be a ball in H, Q ∂B(A, r) and be a geodesic from A to ∈ Q. Then, if A0 is a point of , B(A0, r d(A, A0)) is contained in B(A, r) and Q belongs to ∂B(A , r d(A, A )). 0 0 Proof. of lemma 4.4. Let R be a point in B(A , r d(A, A )), then 0 0 d(R, A) d(R, A0) + d(A0, A) = r.

We now complete the proof of Lemma 4.2. If s is small enough, by Lemma 4.3 there exists an open neighborhood of P such that B(0, s) is contained in B (D , r). Consider now V 0 ∩ V Euc s all Heisenberg balls B(R, ) with R 0. By Lemma 4.4, B(R, ) B(0, s). As 0, B(R, ) shrinks to P , hence B(R, ) is con∈tained in B(0, s) B (D , r) for small→ enough. 0 ∩ V Euc s Clearly, B(R, ) contains P 0 on its boundary. This proves the claim, hence Lemma 4.2.

Let S = ∂ be the boundary of an open C2 set in H. The oriented metric normal to S at P S, P S, is the geodesic arc P S : I H such that ∈ → P S() = whenever I and the interval I is maximal with this property. An equation for S can ∈ P be explicitely be written if we know a parametrization for S. Consider the case ∂tg = 0 at P . Recall the geodesic described in (12) and observe that it points upwards (i.e., the t-co6 ordinate increases with ). Then, P S() = () if is below S in a neighborhood of P in S, while

10 NICOLA ARCOZZI, FAUSTO FERRARI

S() = ( ) if is above S in a neighborhood of P in S. The case of points P where P ∂tg = 0 can be handled as in the Euclidean setting. Theorem 4.1 has a counterpart for the oriented metric normal. Theorem 4.2. Let be an open subset of H with C2 boundary S, in the Euclidean sense, locally dened by the equation g(x, y, t) = 0, with g smooth and such that g = 0 on S. Let P be a non ∇ 6 characteristic point of S. Then the oriented metric normal P S to S at P is a proper subarc of the length-minimizing geodesic having equation u() = 1 Y g(P ) 1 cos 4∂tg(P ) + Xg(P ) sin 4∂tg(P ) 4∂tg g(P ) g(P ) |∇H | |∇H | (14) () = v() = 1 n Xg(P) 1 cos 4∂tg(P ) + Y g(P ) sin 4∂tg(P )o 4∂tg g(P ) g(P )  |∇H | |∇H |  g(P ) 2 4∂ g(P ) 4∂ g(P )  H n t t o s() = 8(|∇∂ g(P ))| 2 g(P ) sin g(P ) t |∇H | |∇H |  n o When ∂tg(P ) = 0, i.e. when the Euclidean plane tangent to S at P is vertical, formula (12) becomes Xg(P ) Y g(P ) (15) () = , , 0 g(P ) g(P ) |∇H | |∇H | P S has the same orientation as , or the opposite one, according to the choice of . If P is characteristic and S is smooth, then the metric normal at P contains P alone.

Next, we give an elementary description of the set of the metric normals’ endpoints. Let S = ∂ be the boundary of an open, connected set in H and assume that S is at least C2. For P S, let Q be endpoint in other than P of the metric normal S (when S ∈ NP NP reduces to the point P , we set Q = P ). The cut-locus in of S (the skeleton of ) is the set KS of such points Q as P varies over S. Below, S refers to the portion of the metric normal at P which lies inside . NP Here are some properties of KS.

Lemma 4.5. Let P S be non characteristic. If P S is not a straight line, then it is a proper subarc of a maximal ∈length minimizing geodesic startingN at P . Proof. Without loss of generality, assume that P = O. Let S = , where is a maximal NP |[0,b] length-minimizing geodesic starting at P and having length a b. If b = a, then (b) = Q belongs to the t-axis, hence R satises length(R) = length() = d(P, Q) = d(Q, S), i.e., for all , R lies on P S. This implies that R˙ (0) is perpendicular to TOS, for all , which is absurd. N Lemma 4.6. Let C be a characteristic point of S. Then, S = C . NC { } Proof. Suppose that Q C S = C is dierent from C. Then, S does not intersect B(Q, d(Q, C)) and meets its closure in∈CN. Hence,{ S} is not smooth at C.

Proposition 4.1. The cut locus KS of S has the following properties. (i) KS has empty interior. (ii) KS contains the characteristic points of S and each characteristic point of S is an accu- mulation point of K S. S Proof. The assertion in (i) is a consequence of the following.

Lemma 4.7. Let Q KS and let be a geodesic from Q to S such that d(Q, S) = length(). Then Q is free∈of points of K . { } S Proof. of the lemma. Let P be the endpoint of in S. Let R be point of KS on , other than Q. By denition of KS, there exists a maximal geodesic through R, having length a > d(R, S),

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP5 11

such that (0) = P S, R = (b), where b = d(R, S) > a, and for b < c a, d((c), S) < c 0 ∈ (otherwise R would not be the endpoint of P 0 S). Clearly, d(R, P 0) = d(R, P ). By the triangle inequality, then N

d(Q, S) d(Q, P 0) d(Q, R) + d(R, P 0) = d(Q, R) + d(R, P ) = d(Q, P ) = d(Q, S) but this is possible only if R lies on the geodesic between Q and P 0, and so Q , contradicting the fact that ’s length does not realize the distance of its points from S past∈ R. This proves the lemma.

The rst assertion in (ii) was proved in Lemma 4.6. Let now C be a characteristic point of S and let Pn be a sequence of noncharacteristic points of S tending to C. (Such sequence exists because the horizontal distribution is not closed under Lie brackets and by Frobenius Theorem). The formula for the geodesic normal in Theorem 4.1 implies that the length of S is no more that d(P , C ), where C is the characteristic point of T S. By continuity, NPn n n n Pn d(Pn, Cn) d(C, C) = 0 as n . Let Rn KS be the endpoint other than Pn of Pn S. Then, → → ∞ ∈ N d(R , C) d(R , P ) + d(P , C) d(R , C ) + d(P , C) 0 n n n n n n n → and this proves (ii).

Proposition 4.2. Let R H KS. Then there is a unique geodesic from R to S such that length = d(R, S). i.e., ther∈ e exists a unique P S such that R S. ∈ ∈ NP Proof. Suppose there are two such geodesics, and 0, having the other endpoint, resp. P and P 0, on S. We can not have P = P 0, otherwise, with a reasoning similar to that of Lemma 4.5, one shows that P is characteristic, hence reduces to a point and R S. Suppose rst that is not a straight line. Let Q K be the endp∈ oint of other ∈ S NP than P . If 0 dos not lie in the (possibly non length-minimizing) prolongment of , then, as in the proof of Proposition 4.1, we have that d(P 0, Q) < d(P, Q) = d(Q, S), a contradiction. The other possibility is that 0 and have in common the arc between Q and R. This leads to a contradiction, too, since it would imply

d(Q, P 0) = d(P 0, R) d(Q, R) < d(P, R) + d(Q, R) = d(P, Q) , hence Q could not belong to . NP If = [P, R] and 0 = [P 0, R] are straight lines, either they lie on the prolongment of each other (but this is not possible, otherwise R KS), or they have dierent directions. In the second case, there exists a point Q such that R∈ [Q, P ] and d(Q, S) = length([P, Q]) (otherwise ∈ R would be the endpoint of P , then R KS) and, as above, we would have d(Q, P 0) < d(P, Q), contradicting the fact that QN . ∈ ∈ NP

A detailed study of the cut-locus KS and of its properties (closedness, Hausdor measure, etcetera) will be the object of further research.

5. Regularity of the distance function near a smooth surface In this section, we prove the regularity of the distance function for a smooth surface S in H and give explicit formulae for its gradient and for its Hessian, restricted to the surface itself. Let S be a non-characteristic smooth surface dened by g(x, y, t) = 0. We consider rst the points of S where ∂ g(x, y, t) = 0. Let t = f(x, y) be the expression of S obtained through the t 6 theorem. Consider the map F : R2 R H R3, → (16) F (u, v, ) = S() N(u,v,f(u,v))

12 NICOLA ARCOZZI, FAUSTO FERRARI

Lemma 5.1. Let be an open subset of H with C2 boundary S, in the Euclidean sense, locally 2 dened by the equation g(x, y, t) = 0, ∂tg(x, y, t) = 0, with g in C . Let F be as in (16) and let P = F (u, v, 0) S. Then, 6 ∈ Xg 1 0 sign(∂tg) g |∇YHg | (17) JF (u, v, 0) = 0 1 sign(∂tg) g  |∇H |  ∂xg ∂yg 2uY g+2vXg sign(∂ g) ∂tg ∂tg t g  |∇H |  hence,   (18) det JF (u, v, 0) = 2d(P, C) If P is characteristic, then det JF (u, v, 0) = 2d(P, C) = 0. The fact that the of the JF (u, v, 0) has an intrinsic geometric meaning can be explained as follows. The 2 3 matrix is expressed with respect to the basis ∂u, ∂v, ∂ of R R, and ∂x, ∂y, ∂t of H R . { } { } The rst basis has an intrinsic meaning. If we change the second basis into X, Y, ∂t , which can be done with a linear transformation having determinant 1, the matrix represen{ ting} JF (u, v, 0) has an invariant form, which is given in the following theorem. Theorem 5.1. Let be an open subset of H with C2 boundary S, in the Euclidean sense, locally dened as g(x, y, t) = 0, with g in C2 and g = 0. Let F be the function dened in (16) and let ∇ 6 P = F (u, v, 0) S, so that ∂tg(P ) = 0. Then, the matrix representing JF (u, v, 0) with respect ∈ 2 6 3 to the basis ∂u, ∂v, ∂ of R R, and X, Y, ∂t of H R is { } { } Xg 1 0 sign(∂tg) g |∇YHg | (19)  0 1 sign(∂tg) g  |∇H | Xg Y g 0  ∂tg ∂tg    Proof. of Lemma 5.1. Suppose that S = (u, v, t) H : t= f(u, v) , { ∈ } We will use the expressions Xf = Xg and Y f = Y g, where g(u, v, t) = t f(u, v). We have (x, y, t) = F (u, v, ) = (u, v, f(u, v)) (x0, y0, t0) 1 where (x0, y0, t0) = u,v() and u,v is the metric normal’s left translate by P ,i.e., by Theorem 4.1, 1 x0 = 4 (Xf sin + Y f(1 cos )) 1 y0 = 4 (Y f sin Xf(1 cos ))  f 2 t = |∇H | ( sin ),  0 8 4 where = f . Since  |∇H | x = u + x0 y = v + y  0 t = f(u, v) + t + 2(vx uy )  0 0 0 we have  xu = 1 + xu0 y = y  u u0 t = f (u, v) 2y + t + 2(vx uy ),  u u 0 u0 u0 u0

 xv = xv0 y = 1 + y  v v0 t = f (u, v) + 2y + t + 2(vx uy )  v v 0 v0 v0 v0 

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP6 13

and

x = x0 y = y  0 t = +t + 2(vx uy ).  0 0 0 We compute the of each coordinate,

4x0 = (Xf) sin + (Y f) (1 cos ) + Xf cos + Y f sin u u u u u = (Xf) sin + (Y f) (1 cos ) u u Xf 4 Xf 4 cos 2Xf(Xf) + 2Y f(Y f) sin 2Xf(Xf) + 2Y f(Y f) , 2 f 3 u u 2 f 3 u u | ∇H | | ∇H | and so (xu0 ) =0 = 0. Analogously | 4x0 = (Xf) sin + (Y f) (1 cos ) v v v 4 4 cos Xf + sin Y f ) Xf(Xf) + Y f(Y f) , f 3 f 3 u u | ∇H | | ∇H | and (yu0 ) =0 = 0; | 4 4 4x0 = Xf cos + Y f sin , f f | ∇H | | ∇H | Xf and (x0 ) =0 = f ; | |∇H | 4y0 = (Y f) sin (Xf) (1 cos ) u u u 4 cos Y f sin Xf Xf(Xf) + Y f(Y f) , f 3 u u | ∇H | and (yu0 ) =0 = 0; | 4y0 = (Y f) sin (Xf) (1 cos ) v v v 4 cos Y f sin Xf Xf(Xf) + Y f(Y f) , f 3 v v | ∇H | and (yv0 ) =0 = 0; | 4 4y0 = cos Y f sin Xf , f 3 | ∇H | Y f and (y0 ) =0 = f . | |∇H | 4 8t0 = 2 Xf(Xf) + Y f(Y f) ( sin ) Xf(Xf) + Y f(Y f) (1 cos ), u u u f u u | ∇H | and (tu0 ) =0 = 0; | 4 8t0 = 2 Xf(Xf) + Y f(Y f) ( sin ) + Xf(Xf) + Y f(Y f) (1 cos ), v v v f v v | ∇H | and (tv0 ) =0 = 0, and, eventually | 8t0 = 4 f (1 cos ), | ∇H | and

t0 = 0.

14 NICOLA ARCOZZI, FAUSTO FERRARI

Hence 1 1 x = 1 + (Xf) sin + (Y f) (1 cos ) u 4 u 4 u 1 Xf 4 1 Xf 4 cos 2Xf(Xf) + 2Y f(Y f) sin 2Xf(Xf) + 2Y f(Y f) 4 2 f 3 u u 4 2 f 3 u u | ∇H | | ∇H | and (xu) =0 = 1; |

xv = xv0

and (xv) =0 = 0; |

x = x0 Xf and (x ) =0 = f ; | |∇H |

yu = yu0

and (yu) =0 = 0; | 1 1 y = 1 + (Y f) sin (Xf) (1 cos ) v 4 v 4 v 1 4 cos Y f sin Xf Xf(Xf) + Y f(Y f) , 4 f 3 v v | ∇H | and (yv) =0 = 1; |

y = y0 , Y f and (y ) =0 f | |∇H | 1 t = f + Xf(Xf) + Y f(Y f) ( sin ) Xf(Xf) + Y f(Y f) (1 cos ) u u 4 u u 2 f u u | ∇H | + 2vx0 2uy0 2y0, u u with (tu) =0 = fu. | 1 t = f (1 cos ) + 2(vx0 uy0 ), 2 | ∇H | Xf Y f and (t ) =0 = 2v f 2u f . | H H Moreover |∇ | |∇ | 1 1 4 t = f + Xf(Xf) + Y f(Y f) ( sin ) Xf(Xf) + Y f(Y f) (1 cos ) v v 4 v v 2 f v v | ∇H | + 2vx0 2uy0 + 2x0 v v and (tv) =0 = fv. As a consequence | Xf 1 0 f x y t |∇YHf | (20) J = 0 1 f u v  H  =0 2vX|∇f 2uY| f fu fv f  H   |∇ |  hence (21) detJF (u, v, 0) = f . | ∇H |

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP7 15

Replacing in the matrix (20) fu and fv by their expressions in terms of g’s derivatives, we obtain Xg 1 0 sign(∂tg) g |∇YHg | (22) JF (u, v, 0) = 0 1 sign(∂tg)  H g  ∂ug ∂vg 2uY|∇g+2|vXg sign(∂ g) ∂tg ∂tg t g  |∇H |  as wished.  

We now consider the set S0 of those points (x0, y0, t0) in S where ∂tg(x0, y0, t0) = 0. We consider two cases. Suppose that (x0, y0, t0) has a neighborhood U in S such that the metric normal at any point of U is a Euclidean straight line, which is normal in the Euclidean sense to S. Hence, in an open neighborhood of (x0, y0, t0), dS is the Euclidean distance, and the smoothness of dS follows. The second case is that where (x0, y0, t0) is the limit of points (x, y, t) of S where ∂tg(x, y, t) = 0. Since g(x , y , t ) = 0, we can assume that, for (x, y, t) in a neighborhhod U of (x , y , t 6), ∇ 0 0 0 6 0 0 0 ∂yg(x, y, t) = 0. By the Implicit Function Theorem, restricting U if necessary, we can assume that g(x, y,6 t) = 0 if and only if y = h(x, t), where h is dened in an open neighborhood of 2 2 (x0, t0) in R . We consider a function H : R R H, dened in an open neighborhood of → (x0, t0, 0),

(23) H(x, t, ) = (x,h(x,t),t)S()

Observe that H is a smooth map, in the Euclidean sense, at points where ∂tg = 0 as well, by (12) and (13). Now, det JH = h JF | | | t | and the latter, when = 0, is equal to g 2 h d(P, C) = h |∇H | | t| | t| ∂ g | t | Since 0 = gyht + gt we have that g |∇H | = h ∂ g |∇H | | y | hence we have an expression analogous to (21), Lemma 5.2. (24) detJH(x, h(x, t), t) = h(x, t) . | ∇H | where Xh = X(h(x, t) y) = hx + 2hht and Y h = Y (h(x, t) y) = 1 2xht. On vertical points P , in particular, det JH(P ) = h2 + 1 > 0 | | x We are now able to prove p k Theorem 5.2. Let S = ∂, being an open C subset of H, k 2, and let dS : H [0, ) be the distance function, → ∞ dS(Q) = d(Q, S) Then, for any non-characteristic point P S there is a neighborhood of P in H such that dS is Ck 1 in . Also, ∈ U U g (25) d = ∇H ∇H S g |∇H | 1 Corollary 5.1. There are C∞ surfaces S such that dS is not even C , in the intrinsec meaning, at its characteristic points.

16 NICOLA ARCOZZI, FAUSTO FERRARI

It suces to consider the plane S = t = 0 . By Theorem 5.2, { } yX xY dS(x, y, 0) = ∇H x2 + y2

which has no limit as (x, y, 0) tends to (0, 0, 0). p

Proof. of Theorem 5.2. Consider rst the case of a portion of S represented by an equation of the type t = f(x, y). The smoothness of dS is an immediate consequence of the Inverse Function Theorem, of the fact that P is non characteristic and (18). We write (x, y, t) = F (u, v, ). We now show (25). Again by the usual Inverse Function Theorem,

1 JF (x, y, t) =0 = 2uY g+2vXg ∂yg Y g | Xg ∂yg Xg + , , g ∂tg g g ∂tg g |∇H | |∇H | |∇H | |∇H | 1 Y g 2uY g+2vXg 2uY g+2vXg ∂yg Y g Y g det(JF (u, v, 0))  g g , g + ∂ g g , g  H H H t H H |∇ 2|uY g+2|∇vXg| ∂yg |∇ | |∇ | |∇ | , , 1  g ∂tg   |∇H |    Hence, the Euclidean gradient of dS on S is

1 2uY g + 2vXg ∂yg d = det(JF (u, v, 0)) , , 1 ∇ S|=0 g ∂ g |∇H | t Then, elementary calculations show that

HdS =0 = (XdS X + Y dS Y ) =0 ∇ | 1 |1 = (JF (u, v, 0)) (∂ g) (Xg X + Y g Y ) t |=0 Now, Lemma 4.1, (11), imply

1 1 2d(P, C) 1 1 (JF (u, v, 0)) (∂ g) = = t g 2d(P, C) g |∇H | |∇H | and (25) follows.

Consider now the case of a characteristic point free portion of S having an equation of U the type y = h(x, t). By Lemma 5.2 and by the Inverse function Theorem, dS is smooth in a neighborhood of . Now, if P Uis an accumulation point of points for which (25) holds, then (25) holds for P as well, by con∈ tinU uity. If there is a neighborhood of P in S such that g (x, y, t) = 0 (i.e., for V t Q , TQS is a vertical plane), then dS is the Euclidean distance near P and Hg = g is the Euclidean∈ V gradient on . Hence, (25) holds by the Euclidean theory. ∇ ∇ V As a byproduct, we have the proof of a special case of a theorem of Monti and Serra-Cassano [13].

2 Corollary 5.2. Let S = ∂ be the boundary of an open C subset of H and let dS be the distance function for S. Then, dS satises the eikonal equation

(26) d = 1, |∇H S| on the surface S.

In the following section we improve on this corollary.

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP8 17

6. The Hessian of the distance function for a smooth surface

Let g be a smooth function from H in R. We dene the horizontal Hessian HessHg and the symmetrized horizontal Hessian Hess g of g. See, e.g., [5], [10] for a discussion of second H order dierential operators in the Heisenberg group. XXg Y Xg (27) Hess g = H XY g Y Y g 1 Let (XY )g = 2 (XY g + Y Xg). XXg (XY ) g (28) Hess g = H (XY ) g Y Y g Theorem 6.1. Let S be a non-characteristic smooth surface in H, dened by the equation t = f(x, y). Let 2 2 Q = XXf (Y f) 2(XY )f Xf Y f + Y Y f (Xf) Then, the horizontal Hessian of dS is (Y f)2 Xf Y f Xf Y f f 5 Q 4 f 3 Y f f 5 Q 4 f 3 (29) Hess d = |∇H | |∇H | |∇H | |∇H | H S  Y f Xf h(Xf)2 Xf Y f i Xf f 5 Q + 4 f 3 f 5 Q + 4 f 3 |∇H | |∇H | |∇H | |∇H |  h i  In particular, the symmetrized horizontal Hessian of the distance function dS, restricted to the surface S, is (Y f)2 Xf Y f Xf Y f (Xf)2 (Y f)2 f 5 Q 4 f 3 f 5 Q + 2 f 3 (30) Hess d = H 2 H 2 H 2 H H S Xf Y|∇f | (Xf)|∇ (Y| f) |∇ (|Xf) |∇Xf Y| f f 5 Q + 2 f 3 f 5 Q + 4 f 3 ! |∇H | |∇H | |∇H | |∇H | and, on S, Q XXf(Y f)2 2(XY ) fXfY f + Y Y f(Xf)2 (31) d = = . H S f 3 f 3 |∇H | | ∇H | 1 In particular, if C is a characteristic point in S, then HessH dS(P ) = O(d(P, C) ) as P C in S. | | → Actually, the formulae hold with g in place of f, if g is a smooth function such that S = g = 0 . { } In order to prove the theorem, we need some preliminary results. 3 We denote by Hess k the Euclidean Hessian of a function k from R in R. If K = (k1, . . . , km) maps R3 in Rm, we write Hess K = (Hess k1, . . . , Hess km) Consider the equation (x, y, t) = F (u, v, ) Let us denote by Hess x, Hess y, Hess t the Euclidean Hessian matrices of x(u, v, ), y(u, v, ), 1 t(u, v, ); let = (u, v, ) and let x, y, y be the coloumns of the Jacobian matrix of F computed when = 0. Let a = + 2y and b = 2x . x t y t Lemma 6.1. The entries of the horizontal Hessian of dS satisy the following relations. (i) We have (32) f XXd = F F HessF (a, a) |∇H | S u| v| Here, c c c denotes the determinant of the 3 3 matrix having columns c , j = 1, 2, 3. 1| 2| 3 j

18 NICOLA ARCOZZI, FAUSTO FERRARI

(ii) We have (33) f Y Y d = F F HessF (b, b) |∇H | S u| v| (iii) XY and Y X satisfy Xd XXd + Y d XY d = 0 (34) S S S S Xd Y Xd + Y d Y Y d = 0 ( S S S S Proof. The Inverse Function Theorem applied to the equation (x, y, t) = F (u, v, ) gives a set of equations for the second derivatives of = dS, w.r.t. x, y, t. Let p, q be any two variables, possibly equal, chosen among x, y, t. Then

xuupq + xvvpq + x pq = Hessx(p, q) (35) y u + y v + y = Hessy( , )  u pq v pq pq p q t u + t v + t = Hesst( , )  u pq v pq pq p q We solve the linear systems with respect to the second partial derivatives of ,  xu, xv, Hessx(p, q) det y , y , Hessy( , )  u v p q  tu, tv, Hesst(p, q) (36) pq =  xu, xv, x  det y , y , y  u v  tu, tv, t . Recall from (11) and (18) that  

xu, xv, x det yu, yv, y = f |∇H | tu, tv, t . =0 Since XX = ∂ + 4y∂ + 4y2∂ , since the determinan t is linear with respect to each column xx xt tt and HessF (U, V ) is bilinear in U and V , then (36) implies then, for = 0, f XXd = F F F XXd |∇H | S u| v| S 2 = Fu Fv [HessF (x, x) + 4yHessF (x, t) + 4y HessF (t, t)] | | = Fu Fv HessF (a, a) | | and this shows (i). (ii) can be proved in the same way, starting from the expression Y Y = ∂ 4x∂ + 4x2∂ . yy yt tt (iii) follows taking derivatives w.r.t. X and Y of the eikonal equation (26).

Proof. of Theorem 6.1. By (i) and (ii) in Lemma 6.1, the expression of XXdS and Y Y dS can be computed if we know HessF restricted to = 0 and J(F 1)(x, y, t) = ( ). By (iii), |=0 x| y| t this allows us to compute XY dS and Y XdS as well. We apply the classical Implicit Function Theorem to (x, y, t) = F (u, v, ). Recall from Lemma 5.1 that the Jacobian of F , Xf 1 0 f x y t |∇YHf | (37) JF (u, v, 0) = J = 0 1 f , u v  H  =0 2vX|∇f 2uY| f fu fv f  H   |∇ | 

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP9 19

whose determinant is detJF (u, v, 0) = f . | ∇H | Its inverse matrix is (38) Xffu Xffv Xf 1 + f 2 f 2 f 2 u v |∇H | |∇H | |∇H | 1 Y ffu 1 + Y ffv Y f J(F )(x, y, t) =0 = J =  f 2 f 2 f 2  = (x y t) | x y t |∇H | |∇H | |∇H | | | =0 fu fv 1 f f f .  H H H   |∇ | |∇ | |∇ |  If g(x, y, t) = t f(x, y), we write XXf = XXg, XY f = XY g, and so on. The following relations will beused below. (Xf) = XXf (Y f) = XY f (39) u u (Xf)v = Y Xf (Y f)v = Y Y f Recalling the derivatives of x, y, t computed in the proof of Lemma 5.1, we can compute 1 1 1 1 x = (Xf) sin + (Y f) (1 cos ) + ( (Xf) cos + (Y f) sin ) uu 4 uu 4 uu 4 u 4 u u 1 Xf 4 + sin 2Xf(Xf) + 2Y f(Y f) 4 u 2 f 3 u u | ∇H | 1 Xf 4 cos 2Xf(Xf) + 2Y f(Y f) ) 4 2 f 3 u u u | ∇H | 1 Xf 4 cos 2Xf(Xf) + 2Y f(Y f) 4 2 f 3 u u | ∇H | 1 Xf 4 sin ( 2Xf(Xf) + 2Y f(Y f) , 4 2 f 3 u u u | ∇H | hence

(xuu) =0 = 0. | 1 1 1 1 x = (Xf) sin + (Y f) (1 cos ) + ( (Xf) cos + (Y f) sin ) vu 4 vu 4 uv 4 u 4 u v 1 Xf 4 + sin 2Xf(Xf) + 2Y f(Y f) 4 v 2 f 3 u u | ∇H | 1 Xf 4 cos ( 2Xf(Xf) + 2Y f(Y f) ) 4 2 f 3 u u v | ∇H | 1 Xf 4 cos 2Xf(Xf) + 2Y f(Y f) v 4 2 f 3 u u | ∇H | 1 Xf 4 sin ( 2Xf(Xf) + 2Y f(Y f) ) 4 2 f 3 u u v | ∇H | (xvu) =0 = 0. | 1 1 x = (Xf) sin + (Y f) (1 cos ) vv 4 vv 4 vv 1 1 + (Xf) cos + (Y f) sin 4 v 4 v 1 4 1 4 ( cos Xf + sin Y f ) Xf(Xf) + Y f(Y f) ) , 4 f 3 4 f 3 u u v | ∇H | | ∇H | (xvv) =0 = 0. |

20 NICOLA ARCOZZI, FAUSTO FERRARI 1 4 1 4 x = (Xf ) cos Xf sin u 4 f u 4 f u | ∇H | | ∇H | 1 4 1 4 + (Y f ) sin + Y f cos , 4 f u 4 u f | ∇H | | ∇H | Y f Y f (xu ) =0 = ((Xf)uY f Xf(Y f)u) = ((XXf)Y f Xf(XY f)). | f 3 f 3 | ∇H | | ∇H | 1 1 x = (Xf ) cos Xf sin f f | ∇H | | ∇H | 1 1 + (Y f ) sin + Y f cos f f | ∇H | | ∇H | 1 1 = Xf sin + Y f cos , f f | ∇H | | ∇H | and 4Y f (x ) =0 = . | f 2 | ∇H | 1 1 x = (Xf ) cos Xf sin v f v v f | ∇H | | ∇H | 1 1 + (Y f ) sin + Y f cos f v v f | ∇H | | ∇H | 1 1 = (Xf ) cos + (Y f ) sin , f v f v | ∇H | | ∇H | and Y f Y f (xv ) =0 = ((Xf)vY f Xf(Y f)v) = ((Y Xf)Y f Xf(Y Y f)). | f 3 f 3 | ∇H | | ∇H | Moreover 1 t = f + ( Xf(Xf) + Y f(Y f) ( sin ) + Xf(Xf) + Y f(Y f) (1 cos )) uu uu 4 u u 2 f u u u | ∇H | + 2vx0 2uy0 2y0 2y0 , uu uu u u so that

(t ) = (f + 2vx0 2uy0 2y0 2y0 ) = f ; uu =0 uu uu uu u u =0 uu analogously

(tvu)=0 = fvu, and

(tvv)=0 = fvv, and

(t ) = 2vx0 2uy0 2y0 , u =0 u u and

(t ) = 2vx0 2uy0 + 2x0 , v =0 v v and

(t ) = 2vx0 2uy0 . =0

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP10 21

Hence

0 0 (XXf)Y f Xf(XY f) Y f Hessx = 0 0 (Y Xf)Y f Xf(Y Y f) . =0 f 3   H (XXf)Y f Xf(XY f) (Y Xf)Y f Xf(Y Y f) 4 f | ∇ | | ∇H | Arguing in the same way we get 

0 0 (XY f)Xf Y f(XXf) Xf Hessy = 0 0 (Y Y f)Xf Y f(Y Xf) . =0 f 3   H (XY f)Xf Y f(XXf) (Y Y f)Xf Y f(Y Xf) 4 f | ∇ | | ∇H |   f f 2vx 2uy 2y uu vu u0 u0 0 Hesst=0 = fvu fvv 2vxv0 2uyv0 + 2x0 . 2vx 2uy 2y 2vx 2uy + 2x 2vx 2uy  u0 u0 0 v0 v0 0 0 0 The horizontal Hessian matrix can be written now directly.  A direct calculation shows that (Y f)2 Xf Y f Xf (40) a = , , ( f 2 f 2 f 2 |∇H | |∇H | |∇H | (viewed as a column vector) and Xf Y f (Xf)2 Y f (41) b = , , f 2 f 2 f 2 |∇H | |∇H | |∇H | We then compute (Y f)2Xf (Xf)2Y f Hess x(a, a) = 2 Q + 4 |=0 f 6 f 4 |∇H | |∇H | (Xf)2Y f (Xf)3 Hess y(a, a) = 2 Q 4 |=0 f 6 f 4 |∇H | |∇H | (Y f)2 Xf Y f (Xf)2 Xf Y f Hess t(a, a) = 2 Q 4 + 8 (vY f + uXf) + 4 (vY f + uXf)Q |=0 f 4 f 2 f 4 f 6 |∇H | |∇H | |∇H | |∇H | Recall now that 1 0 Hessx (a, a) |=0 f XXd = 0 1 Hessy =0(a, a) |∇H | S | 2v Xf 2u Y f Hesst =0(a, a) | (Y f)2 Xf Y f = Q 4 f 4 f 2 |∇H | |∇H | Analogous calculations yield the value of Y Y dS. In order to compute the mixed derivatives, we use (34). Finally, observe that, if S is a C2 surface, then 1 1 Hessd = O( f ) = O(d(P, C) ) | S|=0 |∇H | as P tends to a characteristic point C of S.

The estimate for the Hessian of dS on S cannot be extended outside S. In Section 8 , we see that, for instance, when S is the plane t = 0, the behavior of HessH dS is worst when we approach 0 near the line x = y = 0.

22 NICOLA ARCOZZI, FAUSTO FERRARI

7. Mean Curvature In this section, we make some considerations about the notion of mean curvature for a surface in H (see, e.g.. [9]). Denition 7.1. Let S be a smooth surface in H, dened by the equation g(x, y, t) = 0. The mean curvature of S at P S is ∈ Xg(P ) Y g(P ) (42) h (P ) = X + Y S g(P ) g(P ) |∇H | |∇H | The curvature is well dened for non characteristic points only. Here are some characterizations of mean curvature. The rst is in terms of the distance function and it is an immediate consequence of Theorem 6.1.

Theorem 7.1. Let S = ∂ be the boundary of an open, C2 set in H. Then, for any non characteristic point P in S, (43) d (P ) = h (P ) H S S R. Monti pointed out to us that the same result can be obtained by an explicit calculation, which does not rely on the knowledge of the distance’s Hessian. In [9], it is shown that the mean curvature is the trace of a certain matrix , which is not the Hessian of the distance function. The second is in terms of horizontal curvMes on S. First, we state an immediate consequence of the existence theorem for O.D.E.’s.

Lemma 7.1. Let S be a smooth surface in H and let P S be non-characteristic. Then, modulo restrictions and reparametrizations, there exists a unique∈ horizontal curve through P on S. Suppose that S is dened by g(x, y, t) = 0. A parametrization of the curve is the solution of the O.D.E.’s system ˙ () = Y g(()) X Xg(()) Y (44) (()) (()) ((0) = P Theorem 7.2. Let S be a smooth surface and P S be non-characteristic. Let be the ∈ horizontal curve on S through P , and let p be the projection of H onto the z-plane, p(z, t) = z. 2 Then, hS(P ) is the (mean) curvature of p() in R at p(P ).

Proof. Suppose rst that TP S is a non-vertical plane, that is, that in a neighborhood of P , S admits the parametrization S = f(x, y) = t . The horizontal curve on S through P satises equation (44), hence = (x, y, t){is a solution} of x˙ = Y f() y˙ = Xf()  t˙ = 2yY f() + 2xXf()

Since Xf and Y f are independent of t, we solve for x and y, independently of t, and we nd the dierential equation for = p(),

x˙ = Y f() y˙ = Xf() ( Suppose that Y f = 0 at P (otherwise, we can let Xf = 0 at P , since P is non characteristic). Parametrizing as6 y = (x), we nd 6 Xf y0 = Y f

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP11 23

The curvature of in the Euclidean plane is dened as d y y k = 0 = 00 2 2 3/2 dx 1 + (y0) ! (1 + (y0) ) (Y f)3 d Xf p = 3 Hf dx Y f |∇Y f | = 3 (∂xXf + ∂yXfy0)Y f (∂xY f + ∂yY fy0)Xf Hf |∇ | 2 2 XXf (Y f) 2(XY )f Xf Y f + Y Y f (Xf) = 3 Hf = d = h (P ) |∇ | H S S by Theorem 7.1. We are left with the case when TP S is a vertical plane. If there is an open neighborhood of P in S such that T S is vertical for Q , then a portion of S containing P is a cylinder (i.e.,U Q ∈ U it has an equation of the form ϕ(x, y) = 0 for some C2 function ϕ of the variables x, y), and it is easy to see that, near P , is a curve of the form ϕ(x, y) = 0 (t = t(x, y) Then, = p() has equation ϕ(x, y) = 0. On the other hand, in neighborhood of P in H, dS coincides with the Euclidean distance from S, hence the verication that k = hS(P ) only involves Euclidean considerations which are left to the reader. Suppose now that TP S is a vertical plane, but that in S there is a sequence Pn accumulating

in P , such that TPn S is a non-vertical plane. In the Heisenberg group, vectors belonging to dierent sections of the tangent bundle can be compared by means of left translations. Under the identication H R3, hence of the tangent 3 space at P H with R , the dierential of the left translation by Q = (a, b, c), Q : P Q P , is the matrix∈ L 7→ 1 0 0 d Q = 0 1 0 L  2b 2a 1  which is independent of the point where the dierential iscalculated. By denition of Carnot- Caratheodory distance, d Q maps P in Q P , isometrically. L H H Fix P H and let : I H be a smooth, horizontal curve, such that (0) = P . Let W be a ∈ → section of the horizontal bundle along : W (s) (s) for s I, and W is a smooth map. We dene the of W along at P as ∈ H ∈ H d W (s) d ((0)) ( (s))W (s) W (0) := lim L ds s=0 s 0 s | → We give the notion of derivative of a vector eld with respect to a vector in the Heisenberg group.

Denition 7.2. Let W be a section of the horizontal bundle . Let V P and : I H be H ∈ H → any horizontal curve such that (0) = P and 0(0) = V . We set dH(W )(s) H V W (P ) := . ∇ ds s=0 | Lemma 7.2. Let W = aX + bY be a section of the horizontal bundle and : I H be any H → horizontal curve such that (0) = P and 0(0) = V . Then H W (P ) = V a(P )X(P ) + V b(P )Y (P ) ∇V

24 NICOLA ARCOZZI, FAUSTO FERRARI

where V a and V b are the derivatives of the functions a and b in the direction V , respectively. Proof. Indeed

d ((0)) ( (s))W (s) W (0) L s d ((0)) ( (s)) a((s)), b((s), 22(s)a((s)) 21(s)b((s)) W ((0)) = L s a((s)), b((s), 22(0)a((s)) 21(0)b((s)) W ((0)) = s a((s)) a((0)) b((s)) b((0)) = X((0)) + Y ((0)) s s and as a consequence

d ((0)) ( (s))W ((s)) W ((0)) lim L = V a(P )X(P ) + V b(P )Y (P ) s 0 s →

Let nS(P ) the unit vector normal to S = ∂ at P , in the Heisenberg sense, and pointing outside . If g(x, y, t) = 0 is a local parametrization of S, then Xg(P ) Y g(P ) n (P ) = X + Y S g(P ) P g(P ) P |∇H | |∇H | depending on the orientation of S. Theorem 7.3. The linear operator M S : V H n (P ) maps V S into V S. M S acts on P 7→ ∇V S P P P VP S as multiplication times hS(P ).

Modulo a sign, the map MP S is the Weintgarten map for surfaces in the Heisenberg group.

Proof. If W = nS(P ), we have H n (P ) V S ∇V S ∈ P 2 In fact, dierentiating nS(P ) (s) = 1 with respect to s, nS(P ) = 1(P )XP + 2(P )YP , we obtain | | d d 0 = ((s)) ((s)) + ((s)) ((s)) 1 ds 1 2 ds 2 For s = 0, this relation becomes 0 = n (P ), H n (P ) h S ∇V S iH Xf(P ) Y f(P ) Since nS(P ) = f(P ) XP + f(P ) YP we get, for V = 1X(P ) + 2Y (P ) VP S, |∇H | |∇H | ∈ Xf(P ) Xf(P ) X( ) Y ( ) 1 H H f(P ) H f(P ) (45) V nS(P ) = |∇Y f(P ) | |∇Y f(P ) | ∇ 2 " X( f(P ) ) Y ( f(P ) ) # |∇H | |∇H | H The V V (P ) is dened on the one-dimensional linear space VP S, hence it acts as multiplication7→by∇a constant k. Testing (45) on ( , ) = ( Y f(P ), Xf(P )), i.e. on a 1 2 basis of VP S, we see that Q k = f 3 |∇H | is the mean curvature of S at P .

Observe that multiplication by the matrix in (45) acts on VP S as multiplication by the matrix’ trace.

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP12 25

8. Examples In this section, we see two basic examples of surfaces in H. First, we consider the case of the plane t = 0 . Then, we consider the case of the ball with respect to the Heisenberg distance. In particular,{ } we explicitely compute the Hessian of the function ”distance from a point”. Calculations are easier if we exploit the symmetries of H. Lemma 8.1. Let g be a smooth function on H, which is homogeneous of degree m with respect to the dilation group at the origin. Let Wj be left invariant vector elds on H, j = 1, . . . , n. Then, W . . . W g is homogeneous of degree m n, 1 n m n (46) W1 . . . Wng(P ) = g(P ) In particular, if S is a smooth surface in H and denotes the distance function from S, then , and Hess are homogeneous of degreee 1, 0 and 1, respectively. ∇H H Recall that R denotes both the rotation by around the t-axis in H and the rotation by in span X, Y . We use the same symbol for the matrix { } cos() sin() R = sin() cos() Lemma 8.2. Let f be a smooth function on H, which is invariant under the rotation group at the origin, f(rP ) = f(P ) for in R. Then, and Hess commute with rotations. Namely, ∇H H (47) f(R P ) = R f(P ) ∇H ∇H and

(48) HessHf(RP ) = RHessHf(P ) In (48), the multiplication on the right hand side is a multiplication of 2 2 matrices. 8.1. Nonvertical planes. Let be the plane t = 0 . Observe that all nonvertical planes are isometric to . In Theorem 3.1 we computed the{ metric} normal to at P = (x, y, 0) = (z, 0),

x 2 y 2 2 1 + cos z + 2 sin z | | | | () = (u(), v(), s()) =  y 1 + cos 2 x sin 2 P  2 z 2 z  2 | | | |  z 2 2 | 2| z + sin z | | | |  z The geodesic P realizes the distance from when 2| | . By Lemma 8.1, d is homogeneous of degree 1, d is homogeneous of degree 0 and |Hess| d is homogeneous of degree 1. Since ∇H H d is invariant under rotations around the t-axis, HessHd commutes with the same group of rotations. Let = z ( /2, /2), R = z 0 and [0, 2). Here, we use on H the coordinates | | ∈ | | ∈ (u, v, s) = (R cos cos , R cos sin , R2( + 1/2 sin(2))). Hence cos cos , R cos sin , R sin cos u, v, s J = cos sin , R cos cos , R sin sin R, ,  1 2 2  2R( + 2 sin(2)), 0, 2R cos and   u, v, s detJ = 2R3 cos (cos + sin ). R, ,

26 NICOLA ARCOZZI, FAUSTO FERRARI

As a consequence 2 2 cos cos , cos sin , sin R, , cos + sin cos + sin 2R(cos + sin ) detJ = sin , cos , 0 u, v, s  R cos R cos  cos +sin cos , sin +sin cos , cos  R cos + sin R cos + sin 2R2(cos + sin )  A long, but elementary calculation shows now that, in the new coordinates,  cos cos( ) sin cos + cos sin( ) (49) X = ∂ ∂ ∂ cos + sin R R cos R(cos + sin ) and cos sin( ) cos sin + cos cos( ) (50) Y = ∂ + ∂ ∂ . cos + sin R R cos R(cos + sin ) Hence, Lemma 8.3. Let be the plane t = 0 . Then { } Xd = sin( ) and Y d = cos( ). Proof. Recalling (49) and (50) we get Xd = sin( ) and Y d = cos( ). Remark 8.1. After the computation of Xd, we can deduce the value of Y d keeping in mind (47). Theorem 8.1. Let be the plane t = 0 . Then { } 1 cos( )A, cos( )B Hess d = , H R cos (cos + sin ) sin( )A, sin( )B where A = (sin cos + cos( + ) + cos2 sin( )) and B = (cos cos + sin( + ) + cos2 cos( )). Moreover for P , P = O, then ∈ 6 (d ) = 0, =0 i.e. the mean curvature is 0 Proof. Recalling Lemma 8.3 we get by straightforward computation: sin cos cos sin( ) cos( ) XXd = cos( ) + cos( ) = A R cos R(cos + sin ) R cos

cos sin + cos cos( ) cos( ) Y Xd = cos( ) + cos( ) = B R cos R(cos + sin ) R cos sin cos cos sin( ) sin( ) XY d = sin( ) + sin( ) = A R cos R(cos + sin ) R cos cos sin + cos cos( ) sin( ) Y Y d sin( ) + sin( ) = B. R cos R(cos + sin ) R cos In particular, we have 1 2 cos sin , 2 cos2 Trace(Hess d)=0 = Trace = 0. H R 2 sin2 , 2 cos sin

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP13 27

As a consequence the mean curvature of O is 0. This fact could also be proved as follows. The punctured plane ) is the union ofstraight half line leaving O, hence its Heisenberg curvature at a point P isthe Euclidean curvature of a straight line. by Theorem 7.2.

Remark 8.2. Notice that HessHd is singular as R 0 and as /2. For instance, if = 0, then Y Y d diverges as /2 and R 0. →Indeed → → →

(HessHd)=0 1 ( cos sin ) cos2 , (cos + sin + cos3 ) cos = R cos (cos + sin ) ( cos sin ) sin cos , (cos + sin + cos3 ) sin and tan cos + sin + cos3 tan cos3 (Y Y d ) = = (1 + ). =0 R cos + sin R cos + sin The singularity as R 0 is expected, by Theorem 6.1. The singularity as /2 shows that the estimate at the→ end of Theorem 6.1 can not be extended outside the surface.→

8.2. The Carnot-Charatheodory sphere and the Hessian of the distance function. Here we consider = ∂B(0, 1), the sphere of unitary radius with respect to the Carnot- Charatheodory metric. The poles of are NP = (0, 0, 1/) and SP = (0, 0, 1/). The calculation of the horizontal Hessian of d will be accomplished in several steps. First, we nd the equations of the horizontal curves lying on . Via Theorem 7.2, this allows us to com- pute the mean curvature of at every point. By Theorem 6.1, (29) and (31), and Theorem 7.1,

the second order terms of the formula relating HessH to the equation of are all contained in the expression for the mean curvature of . Hence, in order to compute HessHd we are left with the calculation of some rst order derivatives.

Finally, we note that the calculation of HessHd immediately leads to an expression for the horizontal Hessian of the Carnot-Charatheodory distance in H, Hess d(O, ). H We can parametrize in several useful ways. Let (z, t) . Then, there are [0, 2) and u [ 2, 2] such that ∈ ∈ ∈ i sin(u/2) z(u) = e 2 u (51) u sin(u) (t(u) = 2 u2 The parametric representation in (51) can be easily deduced by the geodesic equations (6). We see, now, how is ruled by horizontal curves. Incidentally, this provides a dierently useful parametrization of .

Proposition 8.1. The surface NP, SP is the disjoint union of horizontal curves (u) = (z(u), t(u)) ( u < 2), { } | | i(h(u/2)+) sin(u/2) z(u) = e 2 u (52) u sin(u) (t(u) = 2 u2 where h satises the conditions h(0) = 0 and

2 cot p (53) h0(p) = cot p p These are the only horizontal curves on .

In (53), observe that h0 is well dened for p = 0 as well.

28 NICOLA ARCOZZI, FAUSTO FERRARI

Proof. First, note that, by (51), any curve parametrized as in (52) lies on . Let now be the curve in (52) corresponding to theta = 0 (by rotation invariance, R = is horizontal if and only if is). The curve = (x, y, t) is horizontal if and only if (54) t˙(u) = 2x˙(u)y(u) 2x(u)y˙(u). Here, sin(u/2) x(u) = cos(h(u/2)) u/2 sin(u/2) (y(u) = sin(h(u/2)) u/2 Let p = u/2 We make (54) into an explicit O.D.E. A calculation shows that sin2(p) 2x˙(2p)y(2p) 2x(2p)y˙(2p) = h0(p) p2 and that sin(p) p cos(p) t˙(2p) = cos(p) p3 Equation (53) follows immediately.

It can be proved that h(p) as p , i.e. that the projection of p() onto the plane t = 0 spiralizes innitely man→ ∞y times→around the origin. It can also be shown that the { } Heisenberg length of , i.e. the Euclidean length of p(), is innite. Next, we compute the mean curvature of . Proposition 8.2. The curvature of at P = (z, t), where sin(u/2) z = 2 u | | u sin(u) (t = 2 u2 is the number u cos u sin u u (55) h(P ) = u cos u sin u 4 sin u 2 2 2 2 Proof. Proof. By Theorem 7.2, we only need to apply the formula for the computation of the mean curvature of a curve in parametric form (see, e.g., [8]) to the curves described in Proposition 8.1.

By Proposition 8.2, in order to compute the Hessian of d, we are left with the calculation of a number of rst order derivatives of f. Lemma 8.4. Let t = f(x, y) be the equation of the upper half of and let (z, t) be as in (51). Then, Xf(z, t) = 4 cos 1 + 2 sin sin(u/2) u u/2 (56) 1 sin(u/2) (Y f(z, t) = 4 sin 2 cos u u/2 Proof. Let t = f(x, y). By the , 1 t t x y (57) J = J J x y u u By (51), t cos(u/2) cos(u/2) J = [sin(u/2) u/2 cos(u/2)] , 0 = h(u/2), 0 u (u/3)3 (u/3)3

METRIC NORMAL AND DISTANCE FUNCTION IN THE HEISENBERG GROUP14 29

where s cos(s) sin(s) h(s) = s2 and 1 sin(u/2) x y h(u/2) cos sin J = 2 u/2 u 1 h(u/2) sin sin(u/2) cos " 2 u/2 # so that 1 sin(u/2) sin(u/2) x y 1 u/2 cos u/2 sin J = 1 1 u h(u/2) sin(u/2) " h(u/2) sin h(u/2) cos # u/2 2 2 Using (57), we obtain t 4 cos 4 sin J = , x y u u and the conclusion of the lemma follows immediately.

We have now all the ingredients for the calculation of HessHd. Its expression is complicated, and we only give a sample of it. Theorem 8.2. Let P = (z, t) be the point of sin(u/2) z(u) = 2 u (58) u sin(u) (t(u) = 2 u2 Then, sin2(u/2) u cos u sin u u u sin(u/2) (59) XXd(P ) = 1 + sin2(u/2) u cos u sin u 4 sin u (1 + sin2(u/2))3/2 2 2 2 2 Proof. It suces to insert in the entry of the HessHdcorresp onding to XXd (see (29)) the 2 expressions for Xf and Y f found in (56) and the expression for h = d = Q/ Hf in |∇ | 2 (55). The equality h = d is a special case of Theorem 7.1, while d = Q/ Hf is (31) in Theorem 6.1. |∇ |

Finally, we observe that if dO(P ) = d(O, P ) is the function measuring the distance from O, and if R = ∂B(O, R), then

HessHdO(P ) = HessHdd(O,P ) (P ) By Lemma 8.1, then, 2 Hess d (P ) = d(O, P ) Hess d ( 1 ). H O H d(O,P ) P Hence, in principle, the horizontal Hessian of the distance function can be computed explicitely.

References [1] N. Arcozzi; D. Morbidelli, work in progress on bi-Lipschitz functions. [2] A. Bellaiche The tangent space in sub-Riemannian geometry in Sub-Riemannian Geometry, Progress in Mathematics, 144. ed. by Bellaiche and J. Risler, Birkhauser Verlag, Basel, 1-78 (1996). [3] G.Citti, M.Manfredini, A.Sarti, Neuronal Oscillations in the Visual Cortex: convergence to the Rie- mannian Mumford-Shah Functional, SIAM Jornal of Mathematical Analysis Volume 35, Number 6, 1394 - 1419. [4] G. Citti, A. Sarti,A cortical based model of perceptual completion in the Roto-Translation space, Preprint. [5] D. Danielli; N. Garofalo; D-M Nhieu,Notions of convexity in Carnot groups. Comm. Anal. Geom. 11 (2003), no. 2, 263–341. [6] B. Franchi; R. Serapioni & F. Serra Cassano,Rectiability and perimeter in the Heisenberg group, Math. Ann. 321, (2001), 479-531. [7] M. Gromov Carnot-Charatheodory spaces seen from within, in Sub-Riemannian Geometry, Progress in Mathematics, 144. ed. by Bellaiche and J. Risler, Birkhauser Verlag, Basel, 79-318 (1996).

30 NICOLA ARCOZZI, FAUSTO FERRARI

[8] D. Gilbarg;N.S. Trudinger, Elliptic partial dierential equations of second order. Second edition. Springer- Verlag, Berlin, 1983. [9] N. Garofalo;S. Pauls The Bernstein problem in the Heisenberg group, preprint (2002). [10] G.Lu; J.J. Manfredi; B. Stroffolini,Convex functions on the Heisenberg group. Calc. Var. Partial Dif- ferential Equations 19 (2004), no. 1, 1–22. [11] R.Montgomery A tour of subriemannian geometries, their geodesics and applications. Mathematical Surveys and Monographs, 91. American Mathematical Society, Providence, RI, 2002. xx+259 pp. [12] R.Monti, Some properties of Carnot-Carathodory balls in the Heisenberg group. Atti Accad. Naz. Lincei Cl. Sci. Fis. Mat. Natur. Rend. Lincei (9) Mat. Appl. 11 (2000), no. 3, 155–167 (2001). [13] R. Monti; F. Serra CassanoSurface measures in Carnot-Carathodory spaces. Calc. Var. Partial Dierential Equations 13 (2001), no. 3, 339–376. [14] A. Nagel; E.M. Stein; & S. Wainger Balls and metrics dened by vector elds I: basic properties, Acta Math. 155, 103-147, (1985). [15] P. Pansu,Une inegalite isoperimetrique sur le groupe de Heisenberg C.R. Acad. Sc. Paris, 295, Serie I, 127-130, (1982) [16] P. Pansu,Metriques de Carnot-Caratheodory et quasiisometries des espaces symetriques de rang un, Ann. of Math., 129, 1-60. [17] A. Sarti, G.Citti, M.Manfrdini, From Neural Oscillations to Variational Problems in the Visual Cortex, Journal of Physiology, Volume 97, Issues 2-3, Pages 379-385 (2003). [18] E.M. Stein, Armonic Analysis: Real variable methods, hortogonality and oscillatory , Princeton University Press, Princeton (1993).

Nicola Arcozzi Dipartimento di Matematica dell’Universita Piazza di Porta S. Donato, 5, 40126 Bologna, Italy E-mail: [email protected]

Fausto Ferrari Dipartimento di Matematica dell’Universita Piazza di Porta S. Donato, 5, 40126 Bologna, Italy and C.I.R.A.M. Via Saragozza, 8, 40123 Bologna, Italy. E-mail: [email protected]