STABLE HYPERSURFACES WITH CONSTANT SCALAR CURVATURE HILARIO´ ALENCAR, WALCY SANTOS AND DETANG ZHOU Abstract. We obtain some nonexistence results for complete non- compact stable hyppersurfaces with nonnegative constant scalar curvature in Euclidean spaces. As a special case we prove that there 4 is no complete noncompact strongly stable hypersurface M in R with zero scalar curvature S2, nonzero Gauss-Kronecker curvature R 3 and finite total curvature (i.e. M jAj < +1). 2000 Mathematics Subject Classification: Primary: 53042. 1. Introduction In this paper we study the complete noncompact stable hypersurfaces with constant scalar curvature in Euclidean spaces. It has been proved by Cheng and Yau [CY] that any complete noncompact hypersurfaces in the Euclidean space with constant scalar curvature and nonnegative sectional curvature must be a generalized cylinder. Note that the as- sumption of nonnegative sectional curvature is a strong condition for hypersurfaces in the Euclidean space with zero scalar curvature since the hypersurface has to be flat in this case. Let M n be a complete ori- n n+1 entable Riemannian manifold and let x : M ! R be an isometric n+1 immersion into the Euclidean space R with constant scalar curvature. We can choose a a global unit normal vector field N and the Riemannian n+1 connections r and re of M and R , respectively, are related by re X Y = rX Y + hA(X);Y iN; where A is the second fundamental form of the immersion, defined by A(X) = −re X N: The authors were partially supported by CNPq and FAPERJ, Brazil. 1 2 Hil´arioAlencar, Walcy Santos and Detang Zhou Let λ1; :::; λn be the eigenvalues of A. The r-mean curvature of the immersion in a point p is defined by 1 X 1 Hr = n λi1 ...λir = nSr; r i1<:::<ir r where Sr is the r-symmetric function of the λ1; :::; λn, H0 = 1 and Hr = 0, for all r ≥ n + 1. For r = 1, H1 = H is the mean curvature of the immersion, in the case r = 2, H2 is the normalized scalar curvature and for r = n, Hn is the Gauss-Kronecker curvature. It is well-known that hypersurfaces with constant scalar curvature are critical points for a geometric variational problem, namely, that associ- ated to the functional Z (1.1) A1(M) = S1 dM; M under compactly supported variations that preserves volume. Let M be a hypersurface in the Euclidean space with constant scalar curva- ture. Following [AdCE], when the scalar curvature is zero, we say that a regular domain D ⊂ M is stable if the critical point is such that 2 d A1 ( dt2 )t=0 ≥ 0, for all variations with compact support in D and when the scalar curvature is nonzero, we say that a regular domain D ⊂ M 2 d A1 is strongly stable if the critical point is such that ( dt2 )t=0 ≥ 0, for all variations with compact support in D. It is natural to study the global properties of hypersurfaces in the Euclidean space with constant scalar curvature. For example we have the following open question (see 4.3 in [AdCE]). 4 Question 1.1. Is there any stable complete hypersurfaces M in R with zero scalar curvature and nonzero the Gauss-Kronecker curvature? We have a partial answer to the question 1.1. Theorem A. (see Theorem 3.1) There is no complete noncompact sta- n+1 ble hypersurface M in R with zero scalar curvature S2 and 3-mean curvature S3 6= 0 satisfying R 3 S1 (1.2) lim BR = 0; R!+1 R2 where BR is the geodesic ball in M. 2 2 When S2 = 0, S1 = jAj we have Stable hypersurfaces with constant scalar curvature 3 Corollary B. There is no complete noncompact stable hypersurface M 4 in R with zero scalar curvature S2, nonzero Gauss-Kronecker curvature R 3 and finite total curvature (i.e. M jAj < +1). We remark that Shen-Zhu (see [SZ]) proved that a complete stable mini- n+1 mal n-dimensional hypersurface in R with finite total curvature must be a hyperplane. The above Corollary can be seen as a similar result in dimension 3 for hypersurfaces with zero scalar curvature. We also prove the following result for hypersurfaces with positive con- stant scalar curvature in Euclidean space. Theorem C. (see Theorem 3.2) There is no complete immersed strongly n n+1 stable hypersurface M ! R , n ≥ 3, with positive constant scalar curvature and polynomial growth of 1-volume, that is R S1dM lim BR < 1; R!1 Rn n where BR is a geodesic ball of radius R of M . As a consequence of the properties of a graph with constant scalar curvature, we have the following corollary: n Corollary D. (see Corollary 4.1) Any entire graph on R with nonneg- ative constant scalar curvature must have zero scalar curvature. This can be compared with a result of Chern [Ch] which says any en- n tire graph on R with constant mean curvature must be minimal. It has been be known by a result of X. Cheng in [Che] (see also [ENR]) n+1 that any complete noncompact stable hypersurface in R with con- stant mean curvature must be minimal if n < 5. It is natural to ask that n+1 any complete noncompact stable hypersurface in R with nonnegative constant scalar curvature must have zero scalar curvature. It should be remarked that Chern[Ch] proved that there is no entire n graph on R with Ricci curvature less than a negative constant. We n don't know whether there exists an entire graph on R with constant negative scalar curvature. The rest of this paper is organized as follows: we include some results and definitions which will be used in the proof of our theorems in Section 2. The proof of main results are given in Section 3 and Section 4 is an appendix in which we prove some stability properties for graphs with constant scalar curvature in the Eucildean space. 4 Hil´arioAlencar, Walcy Santos and Detang Zhou 2. Some stability and index properties for hypersurfaces with S2 = const. We introduce the r'th Newton transformation, Pr : TpM ! TpM, which are defined inductively by P0 = I; Pr = SrI − A ◦ Pr−1; r ≥ 1: The following formulas are useful in the proof (see, [Re], Lemma 2.1). (2.1) trace(Pr) = (n − r)Sr; (2.2) trace(A ◦ Pr) = (r + 1)Sr+1; 2 (2.3) trace(A ◦ Pr) = S1Sr+1 − (r + 2)Sr+2: From [AdCC] we have the second variation formula for hypersurfaces n+1 in a space form of constant curvature c, Qc , with constant 2-mean curvature: 2 d A1 R R 2 1 (2.4) jt=0= hP1(rf);rfidM− (S1S2−3S3+c(n−1)S1)f dM; 8f2C (D): dt2 D D c Definition 2.1. When S2 = 0 and c = 0, M is stable if and only if Z Z 2 (2.5) hP1(rf); rfidM ≥ −3 S3f dM; M M 1 for any f 2 Cc (M). One can see that if P1 ≡ 0, then S3 = 0 and M is stable. When S2 = const: 6= 0, M is stable if and only if Z Z 2 hP1(rf); rfidM ≥ (S1S2 − 3S3 + c(n − 1)S1)f dM; D D 1 R for all f 2 Cc (M) and M fdM = 0. We say that M is strongly stable 1 if and only if the above inequality holds for all f 2 Cc (M). Similar to minimal hypersurface we can also define the index I for hy- persurfaces with constant scalar curvature. Given a relatively compact domain Ω ⊂ M, we denote by Ind 1(Ω) the number of linearly indepen- dent normal deformations with support on Ω that decrease A1. The index of the immersion are defined as 1 1 (2.6) Ind (M) := supfInd (Ω) Ω ⊂ M; Ωrelatively compactg: M is strongly stable if Ind 1(M) = 0. The following result has been known in [El]. Stable hypersurfaces with constant scalar curvature 5 n n+1 Lemma 2.1. Let M ! Qc be a noncompact hypersurface with S2 = const: > 0. If M has finite index then there exist a compact set K ⊂ M such that M n K is strongly stable. For hypersurfaces with constant mean curvature, do Carmo and Zhou [dCZ] proved that n+1 Theorem 2.1. Let x : M n ! M be an isometric immersion with constant mean curvature H. Assume M has subexponential volume growth and finite index. Then there exist a constant R0 such that H ≤ −Ric (N); MnBR0 where N is a smooth normal vector field along M and Ric(N) is the Ricci curvature of M in the normal vector N. The technique in [dCZ] was generalized by Elbert [El] to prove the following result: Theorem 2.2. Let x : M n !Q(c)n+1 be an isometric immersion with 1 S2 = constant > 0. Assume that Ind M < 1 and that the 1-volume of M is infinite and has polynomial growth. Then c is negative and S2 ≤ −c: In particular, it implies that when c = 0 the hypersurfaces in the above theorem must have nonpositive scalar curvature. 3. Proof of the theorems 2 2 When S2 = 0 we know that jS1j = jAj . Thus, if S3 6= 0, we have 2 that jAj > 0. Hence S1 6= 0 and we can choose an orientation such that P1 is semi-positive definite. Since p 2 2 j P1Aj = trace(A ◦ P1) = −3S3; then, when c = 0, M is stable if Z Z p 2 2 (3.1) hP1(rf); rfidM ≥ j P1Aj f dM; M M 1 for any f 2 Cc (M).