Nonexistence and Symmetry of Solutions for Schrodinger¨ Systems Involving Fractional Laplacian

DISCRETE AND CONTINUOUS doi:10.3934/dcds.2019071 DYNAMICAL SYSTEMS Volume 39, Number 3, March 2019 pp. 1595–1611

NONEXISTENCE AND SYMMETRY OF SOLUTIONS FOR SCHRODINGER¨ SYSTEMS INVOLVING FRACTIONAL LAPLACIAN

Ran Zhuo∗ School of mathematics and statistics Huanghuai University Zhumadian, Henan 463000, China Yan Li Department of Mathematics Baylor University Waco, TX 76798, USA

Abstract. In this paper, we consider the following Schrödingersystems involving pseudo-differential operator in Rn ( α (−∆) 2 u(x) = uβ1 (x)vτ1 (x), in Rn, γ (1) (−∆) 2 v(x) = uβ2 (x)vτ2 (x), in Rn, where α and γ are any number between 0 and 2, α does not identically equal to γ. We employ a direct method of moving planes to partial differential equations (PDEs) (1). Instead of using the Caffarelli-Silvestre’s extension method and the method of moving planes in integral forms, we directly apply the method of moving planes to the nonlocal fractional order pseudo-differential system. We obtained radial symmetry in the critical case and non-existence in the subcritical case for positive solutions. In the proof, combining a new approach and the integral definition of the fractional Laplacian, we derive the key tools, which are needed in the method of moving planes, such as, narrow region principle, decay at infinity. The new idea may hopefully be applied to many other problems.

1. Introduction. In recent years, there has been a great deal of interest in using the fractional Laplacian to model diverse physical phenomena, such as anomalous diffusion and quasi-geostrophic flows, turbulence and water waves, molecular dynamics, and relativistic quantum mechanics of stars (see [3],[4],[9],[23]). In particular, the fractional order Laplacian can be understood as the infinitesimal generator of a stable Lévyprocess (see [2]).

2010 Mathematics Subject Classification. Primary: 35A01, 35B50, 35S05; Secondary: 35B09, 35B33. Key words and phrases. Schrödingersystems, fractional Laplacian, narrow region principle, decay at infinity, method of moving planes, Kelvin transform, radial symmetry, nonexistence of positive solutions. The first author is supported by NSFC grant 11701207 and Education Department of Henan Province grant 18B110011. ∗ Corresponding author: Ran Zhuo.

1595 1596 RAN ZHUO AND YAN LI

The fractional Laplacian in Rn is a nonlocal pseudo-differential operator, taking the form Z α/2 u(x) − u(z) (−∆) u(x) = Cn,α PV n+α dz, (2) Rn |x − z| where α is any real number between 0 and 2 and PV stands for the Cauchy principal value. This operator is well defined in S, the Schwartz space of rapidly decreasing C∞ functions in Rn. In this space, it can also be defined equivalently in terms of the Fourier transform (−\∆)α/2u(ξ) = |ξ|αuˆ(ξ), whereu ˆ is the Fourier transform of u. One can extend this operator to a wider space of distributions as the following. Let Z n |u(x)| Lα = {u : R → R | n+α dx < ∞} (see [21]). Rn (1 + |x| ) α/2 For u ∈ Lα, we define (−∆) u as a distribution: < (−∆)α/2u(x), φ > = < u, (−∆)α/2φ >, ∀ φ ∈ S. Throughout the paper, we consider the solutions in this distributional sense. One can verify that, when u is in S, all the above definitions coincide. In this paper, we study the Schrödingersystems (see [5],[18],[20]) involving fractional Laplacian in Rn

α β τ n (−∆) 2 u(x) = u 1 (x)v 1 (x), in R , γ β τ n (3) (−∆) 2 v(x) = u 2 (x)v 2 (x), in R , where α and γ are any real numbers between 0 and 2. We also assume that (n + α)−(n−α)β1 −(n−γ)τ1 ≥ 0, (n+γ)−(n−α)β2 −(n−γ)τ2 ≥ 0, β1 6= β2, τ1 6= τ2, and for i = 1, 2, βi, τi ≥ 1. When α = γ = 2, Li and Ma [17] studied a similar system in the whole space Rn: −∆u(x) = uβ(x)vτ (x), (4) −∆v(x) = vβ(x)uτ (x),

n+2 n+2 for n ≥ 3, 1 ≤ β, τ ≤ n−2 and β + τ = n−2 . When n = 3 and β = 2, τ = 3, Eq. (4) is the stationary Schr¨odingersystem with critical exponents for Bose-Einstein condensate. There they proved

n+2 Proposition 1. (See Li and Ma [17]) Assume that 1 ≤ β < τ ≤ n−2 . Then any 2n 2n L n−2 (Rn) × L n−2 (Rn) radially symmetric solution pair (u, v) to system (4) with critical exponents are unique such that u = v.

In [24] and [25], the authors considered the same Schr¨odingersystem with high n order Laplacian on a upper half space R+ with Navier and Dirichlet boundary conditions:

α  β1 γ1 n (−∆) 2 u(x) = u (x)v (x), in R+,  α  β2 γ2 n  (−∆) 2 v(x) = u (x)v (x), in R+, α −1 n (5) u(x) = −∆u(x) = ··· = (−∆) 2 u(x) = 0, on ∂R+,  α −1 n  v(x) = −∆v(x) = ··· = (−∆) 2 v(x) = 0, on ∂R+, NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1597 and α  β1 γ1 n (−∆) 2 u(x) = u (x)v (x), in R+,  α  β2 γ2 n  (−∆) 2 v(x) = u (x)v (x), in R+,  α −1  ∂u ∂ 2 u n (6) u = = ··· = α −1 = 0, on ∂R+, ∂xn ∂xn 2  α −1  ∂v ∂ 2 v  n  v = = ··· = α −1 = 0, on ∂R+, ∂xn ∂xn 2 n n where R+ is the n-dimensional upper half Euclidean space, R+ = {x = (x1, x2, ··· , n xn) ∈ R |xn > 0}, and β1, γ1, β2, and γ2 satisfy the condition (f1): n+α n n+α 0 ≤ β1, γ1, β2, γ2 ≤ n−α with n−α < β1 + γ1 = β2 + γ2 ≤ n−α , β1 6= β2, γ1 6= γ2. They considered the corresponding integral systems

( R β1 γ1 u(x) = n GN (x, y)u (y)v (y)dy, R+ R β2 γ2 (7) v(x) = n GN (x, y)u (y)v (y)dy, R+ and ( R β1 γ1 u(x) = n GD(x, y)u (y)v (y)dy, R+ R β2 γ2 (8) v(x) = n GD(x, y)u (y)v (y)dy, R+ where 1 1 G (x, y) = c ( − ) N n |x − y|n−α |x∗ − y|n−α is the Green’s function with Navier boundary conditions, and

4xnyn α Z 2 −1 Cn |x−y| z 2 GD(x, y) = n−α n dz, |x − y| 0 (z + 1) 2 is the Green’s function with Dirichlet boundary conditions. Due to the non-locality of the fractional Laplacian with 0 < α < 2, they techni- cally required that α is any even number between 0 and n in PDEs. They proved that the solutions of the corresponding integral equations must satisfy PDEs. Be- cause of technical limitations, they only conjectured that the converse is also true. By using the method of moving planes in integral forms, they veriﬁed

Proposition 2. (See Zhuo, Li and Lv [25])For β1, γ1, β2, and γ2 satisfying (f1), if (u, v) is a pair of non-negative solution of integral systems (7), then u ≡ 0 and v ≡ 0, where α is any real number between 0 and n if n > 3, and 1 < α < n if n = 3.

Proposition 3. (See Zhuo and Li [24]) For β1, γ1, β2, and γ2 satisfying (f1), if p n (u(x), v(x)) is a pair of non-negative solutions of (8), with u, v ∈ Lloc(R+), and n(β1+γ1−1) p = α . Then u(x) ≡ 0 and v(x) ≡ 0. Using the equivalence between PDEs and the corresponding integral equations, they partially proved the nonexistence of solutions for PDEs (5) and (6). For more information of this method, please see [1],[12],[13],[15],[16]. In this paper, we directly work on the nonlocal operator to circumvent the difficulty of equivalence between partial differential equations and integral equations. For fractional Laplacian problems, a useful method is the extension method introduced by Caffarelli and Silverstre (see [6]). By extending the fractional operator to one more dimension, the nonlocal problem becomes a local one taking the form 1598 RAN ZHUO AND YAN LI of a second order elliptic equation. Specifically, let u(x) be a function: Rn → R, and U(x, y): Rn × [0, +∞) → R. Here U(x, y) satisfies the following equation

1−α n+1 div(y ∇U(x, y)) = 0, in R+ , n+1 (9) U(x, 0) = u(x), on ∂R+ , n+1 n where R+ = R × [0, +∞). One can prove

α 1−α ∂U n (−∆) 2 u(x) = −C lim y , in R . y→0+ ∂y In particular, when α = 1, PDE (9) is reduced to the following problem div(y1−α∇U(x, y)) = ∆U(x, y) = 0, in Rn+1, + (10) U(x, 0) = u(x), on ∂Rn+1. For (10), one can express the solution U(x, y) by the Poisson kernel P (x, y): n+1 ∆P (x, y) = 0, in R+ , n+1 (11) P (x, 0) = δ0(x), on ∂R+ , here δ0(x) is δ-function centered at the origin. Then, Z U(x, y) = P (x − ξ, y)u(ξ)dξ. Rn However, the extension method can not be applied to uniformly elliptic nonlocal operators and fully nonlinear nonlocal operators. Moreover, it’s hard to apply this method to system involving fractional Laplacian. In this paper, we employ the method of moving planes directly to the fractional Laplacian, and derive the symmetry and non-existence of solutions for system involving the fractional operator. For more information of the method of moving planes, please see [10],[11],[8] and [19]. Furthermore, this method can be generalized to study the uniformly elliptic nonlocal problem, such as ([22])

Aαu(x) = f(x, u), where Z a(x − y)(u(x) − u(y)) Aαu(x) = Cn,α lim n+α dy, →0 n R \B(x) |x − y|

0 < c0 < a(y) < c1, and hopefully can be applied to equations involving fully nonlinear nonlocal operators. In the paper, we need some key tools, such as the narrow region principle and decay at in inﬁnity to carry out the method of the moving planes. In Section 2, we will accomplish this. In Section 3, we apply the key technical results in the method of moving planes together with a new idea to show that

1,1 1,1 Theorem 1. Assume that u ∈ Lα ∩ Cloc , v ∈ Lγ ∩ Cloc , and τi, βi ≥ 1, i=1,2. If (u, v) is a pair of positive solutions for (3), then (i) in subcritical case, that is, (n + α) − (n − α)β1 − (n − γ)τ1 > 0 and (n + γ) − (n − α)β2 − (n − γ)τ2 > 0,(3) has no positive solution; (ii) in critical case, that is, (n + α) − (n − α)β1 − (n − γ)τ1 = 0 and (n + γ) − (n − α)β2 − (n − γ)τ2 = 0, u and v must be radially symmetric with the same center. NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1599

In above theorem, we prove the results under the weak condition that u ∈ 1,1 1,1 Lα ∩Cloc , v ∈ Lγ ∩Cloc . Because there is no degeneracy assumption on the solution (u(x), v(x)), we need to apply the Kelvin transform. Let 1 x − x0 u¯(x) = u( + x0), |x − x0|n−α |x − x0|2 1 x − x0 v¯(x) = v( + x0) |x − x0|n−γ |x − x0|2 be the Kelvin transform centered at any given point x0. Thenu, ¯ v¯ satisfy

α 1 β τ (−∆) 2 u¯ = u¯ 1 v¯ 1 , |x − x0|(n+α)−(n−α)β1−(n−γ)τ1

γ 1 β τ (−∆) 2 v¯ = u¯ 2 v¯ 2 . |x − x0|(n+γ)−(n−α)β2−(n−γ)τ2

When (n + α) − (n − α)β1 − (n − γ)τ1 > 0 and (n + γ) − (n − α)β2 − (n − γ)τ2 > 0, that is subcritical case, then due to the presence of the singular terms 1 and 1 , we are able to use the |x−x0|(n+α)−(n−α)β1−(n−γ)τ1 |x−x0|(n+γ)−(n−α)β2−(n−γ)τ2 method of moving planes to show thatu ¯ andv ¯ must be radially symmetric about the point x0. Since x0 is an arbitrary point in Rn, we conclude that u, v are constants. This contradicts system (3). This establishes the non-existence of positive solutions. In critical case, (n + α) − (n − α)β1 − (n − γ)τ1 = 0 and (n + γ) − (n − α)β2 − (n − γ)τ2 = 0, we can still utilize the method of moving planes to derive thatu ¯ and v¯ must be radially symmetric about the point x0. Hence u, v are symmetry about some point in Rn. Based on the proof of above theorem, we will investigate the same system in the half space in our next paper. We ﬁrmly believe that we can get similar results in the half space. Together with the results in the whole space and in the half space, we can establish a priori estimates of solutions for a family of nonlocal operators on bounded domains of Euclidean space. In general, let n α X ∂u Lu = (−∆) 2 u + + c(x)u, a ∂x i=1 i α 2 where (−∆)a was introduced in [7] as the uniformly elliptic nonlocal operator,

α Z 2 Cn,α 2u(x) − u(x + y) − u(x − y) (−∆)a u = n+α a(y)dy, 2 Rn |y| and 0 < c0 < a(x) < C0. One can use the Liouville type theorems to obtain a priori estimate for solutions of Lu = f(x, u) with the corresponding boundary conditions.

2. Key tools in the method of moving planes. In the section, we will show the key ingredients in the method of moving planes, such as narrow region principle and decay at inﬁnity. First we introduce some basic notation needed in the method of moving planes. For a given real number λ, denote n Σλ = {x = (x1, x2, ··· , xn) ∈ R |x1 ≤ λ} , 1600 RAN ZHUO AND YAN LI

λ Σfλ = x |x ∈ Σλ , n Tλ = {x ∈ R |x1 = λ} , and let λ x = (2λ − x1, x2, ··· , xn) be the reﬂection of the point x = (x1, x2, ··· , xn) about the plane Tλ.

Theorem 2.1. (Narrow Region Principle) Let Ω ⊆ {x|λ − l < x1 < λ} be a 1,1 bounded narrow region in Σλ for l > 0 small. Assume that U ∈ Lα ∩ Cloc (Ω) and 1,1 V ∈ Lγ ∩ Cloc (Ω) are lower semi-continuous on Ω. If bi(x) and ci(x) are positive and bounded from below in Ω, i = 1, 2,  α (−∆) 2 U(x) ≥ b1(x)U(x) + c1(x)V (x), in Ω,  γ  (−∆) 2 V (x) ≥ b (x)U(x) + c (x)V (x), in Ω, 2 2 (12)  U(x) ≥ 0,V (x) ≥ 0, in Σλ \ Ω,  λ λ  U(x ) = −U(x),V (x ) = −V (x) in Σλ, then for suﬃciently small l, we get U(x) ≥ 0,V (x) ≥ 0, ∀x ∈ Ω. (13) Furthermore, if U(x) = 0 and V (x) = 0 at some point in Ω, then U(x) = 0,V (x) = 0, a.e. x ∈ Rn. For an unbounded narrow region Ω, if we suppose lim U(x) ≥ 0, lim V (x) ≥ 0, |x|→∞ |x|→∞ the above conclusions also hold. Proof of Theorem 2.1. If (13) does not hold, by the lower semi-continuity of U(x) and V (x) on Ω, there exist some points x0, x1 ∈ Ω, such that U(x0) = min U(x) < 0,V (x1) = min V (x) < 0. x∈Ω x∈Ω Actually, by (12), one can further deduce that x0 and x1 are in the interior of Ω. By the elementary calculation, we derive 0 α Z U(x ) − U(y) 2 0 (−∆) U(x ) = Cn,α PV 0 n+α dy Rn |x − y| Z U(x0) − U(y) Z U(x0) − U(y) = C PV [ dy + dy] n,α |x0 − y|n+α |x0 − y|n+α Σλ Σfλ Z U(x0) − U(y) Z U(x0) − U(yλ) = Cn,α PV [ 0 n+α dy + 0 λ n+α dy] Σλ |x − y| Σλ |x − y | Z U(x0) − U(y) Z U(x0) + U(y) = Cn,α PV [ 0 n+α dy + 0 λ n+α dy] Σλ |x − y| Σλ |x − y | Z U(x0) − U(y) + U(x0) + U(y) ≤ Cn,α PV 0 λ n+α dy Σλ |x − y | Z 2U(x0) ≤ Cn,α PV 0 λ n+α dy. (14) Σλ |x − y | NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1601

0 n 0 0 0 0 0 Set H = {y = (y1, y ) ∈ R |l < y1 −x1 < 1, |y −(x ) | < 1}, and let ρ = y1 −x1, t = |y0 − (x0)0|, Z 1 Z 1 0 λ n+α dy ≥ 0 n+α dy Σλ |x − y | H |x − y| Z 1 Z 1 n−2 ωn−1t dt = n+α dρ. l 0 (ρ2 + t2) 2 Let t = ρs, it follows from the above inequality tht 1 1 Z 1 Z Z ρ ω (ρs)n−2ρds dy ≥ n−1 dρ 0 λ n+α n+α 2 n+α Σλ |x − y | l 0 ρ (1 + s ) 2 Z 1 Z 1 n−2 1 ωn−1s ≥ 1+α dρ n+α ds l ρ 0 (1 + s2) 2 C ≥ , (15) lα where ωn−1 is the area of (n − 1)-dimensional unit sphere, and C denotes some constant. By (14) and (15), we derive

α 0 C 0 (−∆) 2 U(x ) ≤ U(x ). (16) lα Similarly, we get

γ 1 C 1 (−∆) 2 V (x ) ≤ V (x ). (17) lγ By the first inequality of (12) and (16), one can see that there exists some constant a1 > 0, such that for sufficiently small l we have 1 a U(x0) ≥ c (x0)V (x0). (18) 1 lα 1 Similarly, there exists some constant a2 > 0 such that for sufficiently small l, we have 1 a V (x1) ≥ b (x1)U(x1). (19) 2 lγ 2 Combining (18) with (19), it gives 1 a U(x0) ≥ c (x0)V (x1) ≥ a c (x0)lγ b (x1)U(x0), (20) 1 lα 1 2 1 2 that is 1 α+γ 0 1 ≤ 1. a2l c1(x )b2(x ) It’s trivial that the inequality does not hold for sufficiently small l. Hence (13) must be true.

Theorem 2.2. (Decay at Inﬁnity) Let Ω be an unbounded region in Σλ. Assume 1,1 1,1 that U ∈ Lα ∩ Cloc (Ω), V ∈ Lγ ∩ Cloc (Ω), (U, V ) satisfy the following equations  α (−∆) 2 U(x) ≥ b1(x)U(x) + c1(x)V (x), in Ω,  γ  (−∆) 2 V (x) ≥ b (x)U(x) + c (x)V (x), in Ω, 2 2 (21)  U(x) ≥ 0,V (x) ≥ 0, in Σλ \ Ω,  λ λ  U(x ) = −U(x),V (x ) = −V (x) in Σλ, 1602 RAN ZHUO AND YAN LI where 1 1 b (x) ∼ , c (x) ∼ , as |x| → ∞, (22) 1 |x|2α 1 |x|α+γ 1 1 b (x) ∼ , c (x) ∼ , as |x| → ∞, (23) 2 |x|α+γ 2 |x|2γ and bi(x), ci(x) are nonnegative in Ω, i = 1, 2. Then, there exists a constant R0 > 0, which depends on bi(x) and ci(x), i = 1, 2, but is independent of U(x) and V (x), such that if U(x0) = min U(x) < 0,V (x1) = min V (x) < 0, x∈Ω x∈Ω then 0 1 |x | ≤ R0, or |x | ≤ R0. Proof of Theorem 2.2. Similar to (14), we have 0 α Z 2U(x ) 2 0 (−∆) U(x ) ≤ Cn,α PV 0 λ n+α dy. (24) Σλ |x − y | For ﬁxed λ, when |x0| ≥ λ and |x1| ≥ λ, it’s easy to derive Z 1 C 0 λ n+α dy ≥ 0 α , (25) Σλ |x − y | |x | here C denotes some constant. Combining (24) with (25),

α 0 0 1 (−∆) 2 U(x ) ≤ CU(x ) . (26) |x0|α Similarly, we get

α 1 1 1 (−∆) 2 V (x ) ≤ CV (x ) . (27) |x1|γ It follows from the ﬁrst inequality of (21) and (26) that 1 CU(x0) ≥ c (x0)V (x0) + b (x0)U(x0). |x0|α 1 1

Combining this with degenerate assumption of b1(x), it’s easy to see that there exists some constant C1 > 0 such that C 1 U(x0) ≥ c (x0)V (x0). (28) |x0|α 1

Similarly, there exists some constant C2 > 0 such that C 2 V (x1) ≥ b (x1)U(x1). (29) |x1|γ 2 From (28) and (29), we get C 1 U(x0) ≥ c (x0)V (x1) ≥ c (x0)|x1|γ b (x1)U(x1) |x0|α 1 1 2 0 1 γ 1 0 ≥ c1(x )|x | b2(x )U(x ). That is C1 0 α 1 γ 0 1 ≤ 1. |x | |x | c1(x )b2(x ) NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1603

However, for |x0| and |x1| suﬃciently large, the inequality above is not true. There- fore, there exists R0 > 0 such that 0 1 |x | ≤ R0, or |x | ≤ R0. This completes the proof.

3. Rotational symmetry of solutions for Schr¨odingersystem. In this section, we give the proof of Theorem 1. Because these is no decay conditions on u and v in inﬁnity, we apply the Kelvin transform. For any z0 ∈ Rn, consider the Kelvin transform centered at z0 1 x − z0 u¯(x) = u( + z0), (30) |x − z0|n−α |x − z0|2 1 x − z0 v¯(x) = v( + z0). (31) |x − z0|n−γ |x − z0|2 For simplicity of arguments, we will only show the case when z0 is the origin, while the proof for a general z0 is entirely similar. Let 1 x u¯(x) = u( ), (32) |x|n−α |x|2 1 x v¯(x) = v( ) (33) |x|n−γ |x|2 be the Kelvin transform of u and v centered at the origin. It’s easy to see 1 1 u¯(x) ∼ , v¯(x) ∼ , for large |x|. (34) |x|n−α |x|n−γ It is well known that

α 1 α x (−∆) 2 u¯(x) = ((−∆) 2 u)( ) |x|n+α |x|2 1 x x = uβ1 ( )vτ1 ( ) |x|n+α |x|2 |x|2 1 = |x|(n−α)β1 |x|(n−γ)τ1 u¯β1 (x)¯vτ1 (x) |x|n+α 1 = u¯β1 (x)¯vτ1 (x), (35) |x|a1 where a1 = (n + α) − (n − α)β1 − (n − γ)τ1. Similarly, we have

γ 1 β τ (−∆) 2 v¯(x) = u¯ 2 (x)¯v 2 (x), (36) |x|a2 here a2 = (n + γ) − (n − α)β2 − (n − γ)τ2. Let λ x = (2λ − x1, x2, ··· , xn) be the reﬂection of the point x = (x1, x2, ··· , xn) about the plane Tλ. From (35) and (36), it’s easy to derive

α λ 1 β λ τ λ (−∆) 2 u¯(x ) = u¯ 1 (x )¯v 1 (x ), (37) |xλ|a1 1604 RAN ZHUO AND YAN LI

γ λ 1 β λ τ λ (−∆) 2 v¯(x ) = u¯ 2 (x )¯v 2 (x ), (38) |xλ|a2 where a1 = (n + α) − (n − α)β1 − (n − γ)τ1, a2 = (n + γ) − (n − α)β2 − (n − γ)τ2.

3.1. System in subcritical case. In the subcritical case, a1 = (n + α) − (n − α)β1 − (n − γ)τ1 > 0 and a2 = (n + γ) − (n − α)β2 − (n − γ)τ2 > 0, we show that (3) has no positive solution. Proof. Let λ λ Uλ(x) =u ¯(x ) − u¯(x),Vλ(x) =v ¯(x ) − v¯(x).

By the deﬁnition of Uλ and Vλ, we have

lim Uλ(x) = 0, lim Vλ(x) = 0. |x|→∞ |x|→∞

This implies that Uλ and Vλ attain negative minimum in the interior of Σλ. Deﬁne u v Σλ = {x ∈ Σλ|Uλ(x) < 0} , Σλ = {x ∈ Σλ|Vλ(x) < 0} . The proof consists of two steps. Step 1. We will show that, for λ suﬃciently negative, λ Uλ(x) ≥ 0,Vλ(x) ≥ 0, x ∈ Σλ \{0 }. (39) u v By an elementary calculation, we derive that, for x ∈ Σλ ∩ Σλ,

α 1 1 β1 λ τ1 λ β1 τ1 (−∆) 2 Uλ(x) = u¯ (x )¯v (x ) − u¯ (x)¯v (x) |xλ|a1 |x|a1 1 1 = u¯β1 (xλ)¯vτ1 (xλ) − u¯β1 (xλ)¯vτ1 (xλ) |xλ|a1 |x|a1 1 1 + u¯β1 (xλ)¯vτ1 (xλ) − u¯β1 (x)¯vτ1 (x) |x|a1 |x|a1 1 1 1 = ( − )¯uβ1 (xλ)¯vτ1 (xλ) + (¯uβ1 (xλ)¯vτ1 (xλ) − u¯β1 (x)¯vτ1 (x)) |xλ|a1 |x|a1 |x|a1 1 ≥ (¯uβ1 (xλ)¯vτ1 (xλ) − u¯β1 (x)¯vτ1 (x)) |x|a1 1 ≥ (¯uβ1 (xλ)¯vτ1 (xλ) − u¯β1 (xλ)¯vτ1 (x) +u ¯β1 (xλ)¯vτ1 (x) − u¯β1 (x)¯vτ1 (x)) |x|a1 1 ≥ [¯uβ1 (xλ)(¯vτ1 (xλ) − v¯τ1 (x)) +v ¯τ1 (x)(¯uβ1 (xλ) − u¯β1 (x))] |x|a1 For the above inequality, applying the Mean Value Theorem, α (−∆) 2 Uλ(x) c ≥ [¯uβ1 (xλ)¯vτ1−1(ξ)(¯v(xλ) − v¯(x)) +v ¯τ1 (x)¯uβ1−1(η)(¯u(xλ) − u¯(x))] |x|a1 c ≥ [¯uβ1 (xλ)¯vτ1−1(x)(¯v(xλ) − v¯(x)) +v ¯τ1 (x)¯uβ1−1(x)(¯u(xλ) − u¯(x))] |x|a1 c c β1 λ τ1−1 τ1 β1−1 ≥ u¯ (x )¯v (x)Vλ(x) + v¯ (x)¯u (x)Uλ(x) |x|a1 |x|a1

≥ c1(x)Vλ(x) + b1(x)Uλ(x),

λ c β1 λ τ1−1 where ξ and η are valued between x and x, c1(x) = |x|a1 u¯ (x )¯v (x), b1(x) = c τ1 β1−1 |x|a1 v¯ (x)¯u (x). NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1605

That is, α (−∆) 2 Uλ(x) ≥ c1(x)Vλ(x) + b1(x)Uλ(x). (40) By (34), it is easy to derive that 1 1 c (x) ∼ , b (x) ∼ , for suﬃciently large |x|. (41) 1 |x|α+γ 1 |x|2α Similarly, γ (−∆) 2 Vλ(x) ≥ c2(x)Vλ(x) + b2(x)Uλ(x), (42)

c β2 λ τ2−1 c τ2 β2−1 where c2(x) = |x|a2 u¯ (x )¯v (x), b2(x) = |x|a2 v¯ (x)¯u (x). By (34), we have 1 1 c (x) ∼ , b (x) ∼ , for suﬃciently large |x|. (43) 2 |x|2γ 2 |x|α+γ Suppose there exists some point x0 such that 0 Uλ(x ) = min Uλ(x) < 0. x∈Σλ Similar to (16), for |x0| > λ, we get

α 0 0 1 (−∆) 2 U (x ) ≤ cU (x ) . (44) λ λ |x0|α Combining this with (40), we deduce 1 cU (x0) ≥ c (x0)V (x0) + b (x0)U (x0). (45) λ |x0|α 1 λ 1 λ

By the degeneracy of b1(x) at inﬁnity and (45), for suﬃciently negative λ, c U (x0) ≥ c (x0)V (x0). (46) |x0|α λ 1 λ Next we suppose that there is some point x1 such that V (x1) = min V (x) < 0. λ v λ x∈Σλ Similar to (44), we obtain

γ 1 1 1 (−∆) 2 V (x ) ≤ cV (x ) . (47) λ λ |x1|γ Combining (42) and (47), 1 cV (x1) ≥ c (x1)V (x1) + b (x1)U (x1). (48) λ |x1|α 2 λ 2 λ

From the degeneracy of c2(x) at inﬁnity and (48), for suﬃciently negative λ, we have c V (x1) ≥ b (x1)U (x1). (49) |x1|γ λ 2 λ Combining (46) with (49), we derive that c V (x1) ≥ b (x1)U (x0) ≥ b (x1)|x0|αc (x0)V (x0) |x1|γ λ 2 λ 2 1 λ 1 0 α 0 1 ≥ b2(x )|x | c1(x )Vλ(x ).

Using the degeneracy of b2(x) and c1(x) at inﬁnity, we arrive at c 1 1 V (x1) ≥ V (x1). |x1|γ λ |x1|α+γ |x0|γ λ 1606 RAN ZHUO AND YAN LI

That is, c ≥ 1. |x1|α|x0|γ For sufficiently negative λ, the inequality does not hold. From Theorem 2.2( Decay at Infinity), for sufficiently negative λ (or |λ| < R0 in Theorem 2.2), at least one of Uλ and Vλ is greater than or equal to 0. Without loss of generality, we assume that λ Uλ(x) ≥ 0, x ∈ Σλ \{0 }. (50)

To prove (50) also holds for Vλ, we argue by contradiction. λ If Vλ is negative somewhere in Σλ \{0 }, then there must exist somex ¯ ∈ Σλ such that

Vλ(¯x) = min Vλ(x) < 0. x∈Σλ From previous arguments of (42) and (47), we know that

1 γ 0 > cV (¯x) ≥ (−∆) 2 V (¯x) λ |x¯|γ λ

≥ c2(¯x)Vλ(¯x) + b2(¯x)Uλ(¯x),

c β2 λ τ2−1 c τ2 β2−1 here c2(x) = |x|a2 u¯ (x )¯v (x), b2(x) = |x|a2 v¯ (x)¯u (x). For the above inequality, combining with (49), we derive that

γ 0 > (−∆) 2 Vλ(¯x) γ ≥ cb2(¯x)c2(¯x)|x¯| Uλ(¯x) + b2(¯x)Uλ(¯x) ≥ 0 This is a contradiction. And we complete step 1.

Step 2. Step 1 provides a starting point for us to move the plane Tλ to the right along x1 direction as long as inequality (39) holds. Deﬁne

µ λ0 = sup{λ < 0|Uµ(x) ≥ 0,Vµ(x) ≥ 0, ∀x ∈ Σµ \{0 }, µ ≤ λ}. In the step, we will prove that

λ0 = 0, (51) and

λ0 Uλ0 (x) ≡ 0,Vλ0 (x) ≡ 0, ∀x ∈ Σλ0 \{0 }. (52) Suppose that

λ0 < 0, we will show that the plane Tλ can be moved further more. That is, there exists some small > 0, such that for any λ ∈ (λ0, λ0 + ), we have λ Uλ(x) ≡ 0,Vλ(x) ≡ 0, ∀x ∈ Σλ \{0 }. (53)

This is a contradiction with the deﬁnition of λ0. Therefore, we derive that

λ0 = 0.

Actually, for λ0 < 0, λ Uλ0 (x) > 0,Vλ0 (x) > 0, ∀x ∈ Σλ0 \{0 }. (54) NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1607

Otherwise, at least one of Uλ0 (x) and Vλ0 (x) is greater than or equal to zero.

Without loss generality, we may assume that Uλ0 (x) ≥ 0. That is, there exists some pointx ˆ such that

Uλ0 (ˆx) = min Uλ0 (x) = 0. Σλ0 It follows that

α Z −Uλ0 (y) 2 (−∆) Uλ0 (ˆx) = Cn,α PV n+α dy Rn |xˆ − y| Z Z −Uλ0 (y) −Uλ0 (y) = Cn,α PV [ n+α dy + n+α dy] |xˆ − y| n |xˆ − y| Σλ0 R \Σλ0 Z Z −Uλ0 (y) Uλ0 (y) = Cn,α PV [ dy + dy] |xˆ − y|n+α |xˆ − yλ0 |n+α Σλ0 Σλ0 Z 1 1 = Cn,α PV ( − )Uλ (y)dy |xˆ − yλ0 |n+α |xˆ − y|n+α 0 Σλ0 ≤ 0. (55) On the other hand,

β1 λ0 τ1 λ0 β1 τ1 α u¯ (ˆx )¯v (ˆx ) u¯ (ˆx)¯v (ˆx) (−∆) 2 Uλ (ˆx) = − 0 |xˆλ0 |a1 |xˆ|a1 u¯β1 (ˆx)¯vτ1 (ˆxλ0 ) u¯β1 (ˆx)¯vτ1 (ˆx) = − |xˆλ0 |a1 |xˆ|a1 v¯τ1 (ˆxλ0 ) − v¯τ1 (ˆx) =u ¯β1 (ˆx)[ |xˆ|a1 1 1 + ( − )¯vτ1 (ˆx)] |xˆλ0 |a1 |xˆ|a1 > 0 This is a contradiction with (55). Hence (54) holds.

From [26], we have the integral expressions of Uλ0 and Vλ0 . Combining it with the proof of Appendix A in [14], we can show that there exists c0 > 0 such that, for suﬃciently small ,

λ0 λ0 Uλ0 (x),Vλ0 ≥ c0, ∀x ∈ B(0 ) \{0 }. (56) Together with the above bounded-away-from-0 result, we derive that for δ > 0, there exists some constant c0 > 0 such that

λ0 Uλ0 (x),Vλ0 ≥ c0, ∀x ∈ (Σλ0−δ \{0 }) ∩ BR0 (0). (57)

For , δ |λ0|, λ0 λ 0 ∈ (Σλ0−δ\{0 0 }) ∩ BR0 (0). Since Uλ and Vλ depend on λ continuously, we have

λ0 Uλ0 (x),Vλ0 ≥ 0, ∀x ∈ (Σλ0−δ \{0 }) ∩ BR0 (0). (58) By Theorem 2.2(Decay at inﬁnity), we know that if

Uλ(ˆx) = min Uλ < 0, Σλ then there exists a large R0 such that

|xˆ| ≤ R0. 1608 RAN ZHUO AND YAN LI

Hencex ˆ ∈ (Σλ \ Σλ0−δ) ∩ BR0 (0). For suﬃciently large R0, similar to (46), we obtain

Vλ(ˆx) < 0. Therefore, there exists some pointx ¯ such that

Vλ(¯x) = min Vλ < 0. Σλ Ifx ¯ ∈ BC ∩ Σ , similar to (49), R0 λ cV (¯x) 0 > λ ≥ b (¯x)U (¯x). (59) |x¯|γ 2 λ

Meanwhile, for Uλ atx ˆ, similar to (46), we get cU (ˆx) λ ≥ c (ˆx)V (ˆx). (60) | + δ|γ 1 λ By (59) and (60), we have γ γ b2(¯x)|x¯| c1(ˆx)| + δ| ≥ c. (61) γ Using the degeneracy of c1, we know that c1(ˆx) is bounded. Notice that b2(¯x)|x¯| is also bounded for |x¯| > R0. Hence for δ suﬃciently small, (61) does not hold. This impliesx ¯ ∈ BC ∩ Σ will not happen. R0 λ Employing Theorem 2.1( Narrow region principle), let the narrow region Ω =

(Σλ \ Σλ0−δ) ∩ BR0 (0), while Uλ and Vλ satisfy system (12), we obtain

Uλ(x) ≥ 0,Vλ(x) ≥ 0, ∀x ∈ (Σλ \ Σλ0−δ) ∩ BR0 (0).

Now we conclude that neither Uλ(x) nor Vλ(x) has negative minimum in Σλ \ {0λ}. Hence, λ Uλ(x),Vλ(x) ≥ 0, ∀x ∈ Σλ \{0 }. This completes the proof of (53). Thus we obtain

λ0 = 0.

Similarly, we can move the plane from x1 = +∞ near to the left, and we can show that λ Uλ(x),Vλ(x) ≤ 0, ∀x ∈ Σλ \{0 }. Therefore we conclude that

λ0 λ0 = 0,Uλ0 (x) ≡ 0,Vλ0 (x) ≡ 0, ∀x ∈ Σλ0 \{0 }.

Since the direction of x1-axis is arbitrary, we derive thatu ¯ andv ¯ are radially symmetric about the origin. For any point z0 ∈ Rn, applying the Kelvin transform centered at z0 (see (30),(31)), and by an entirely similar argument, one can show thatu ¯ andv ¯ are radially symmetric about z0. Let z1 and z2 be any two points in Rn, and we choose the coordinate system so 0 z1+z2 that the midpoint z = 2 is the origin. Sinceu ¯ andv ¯ are radially symmetric about z0, we have u(z1) = u(z2) and v(z1) = v(z2). This implies that u and v must be constants. Positive constant solutions do not satisfy system (3). That is, in subcritical case, there is no positive solution for system (3). NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1609

3.2. System in critical case. In critical case, a1 = (n+α)−(n−α)β1−(n−γ)τ1 = 0 and a2 = (n + γ) − (n − α)β2 − (n − γ)τ2 = 0. In this section, we still utilize the Kelvin transform of u and v centered at the origin (see (32),(33)). We will show that eitheru ¯ andv ¯ are symmetric about the origin oru ¯ andv ¯ are symmetric about some point. We still use the notations introduced in the subcritical case. The argument is quite similar to, but not entirely the same as that in the subcritical case. Hence we still present some details here. Proof. In critical case, similar to (40), we get α 2 ¯ (−∆) Uλ(x) ≥ c¯1(x)Vλ(x) + b1(x)Uλ(x), (62)

¯ τ1 β1−1 β1 λ τ1−1 here b1 =v ¯ (x)¯u (x),c ¯1(x) =u ¯ (x )¯v (x). Similar to (42), we have γ 2 ¯ (−∆) Vλ(x) ≥ c¯2(x)Vλ(x) + b2(x)Uλ(x), (63)

¯ τ2 β2−1 β2 λ τ2−1 where b2(x) =v ¯ (x)¯u (x),c ¯2(x) =u ¯ (x )¯v (x). By (34), it is easy to derive that 1 1 c (x) ∼ , b (x) ∼ , for large |x|. (64) 1 |x|α+γ 1 |x|2α 1 1 c (x) ∼ , b (x) ∼ , for lager |x|. (65) 2 |x|2γ 2 |x|α+γ The remaining proof is the same as that in the subcritical case. We can show that, for λ suﬃciently negative, λ Uλ,Vλ ≥ 0, x ∈ Σλ\{0 }. Deﬁne µ λ0 = sup{λ ≤ 0|Uµ(x) ≥ 0,Vµ(x) ≥ 0, ∀x ∈ Σµ{0 }, µ ≤ λ}. We consider two possible cases.

Case (i). λ0 < 0. For λ0 < 0, either

λ0 Uλ0 (x) = Vλ0 (x) ≡ 0, x ∈ Σλ0 \{0 }, (66) or λ0 Uλ0 (x),Vλ0 (x) > 0, x ∈ Σλ0 \{0 }. (67)

We suppose that there exists some pointx ˆ ∈ Σλ0 such that

Uλ0 (ˆx) = min Uλ0 = 0, x∈Σλ0 then

Uλ0 (x) ≡ 0, x ∈ Σλ0 . (68) Otherwise,

α Z −Uλ0 (y) 2 (−∆) Uλ0 (ˆx) = Cn,α PV n+α dy < 0. Rn |xˆ − y| On the other hand, α 2 β1 λ0 τ1 λ0 β1 τ1 (−∆) Uλ0 (ˆx) =u ¯ (ˆx )¯v (ˆx ) − u¯ (ˆx)¯v (ˆx) =u ¯β1 (ˆxλ0 )(¯vτ1 (ˆxλ0 ) − v¯τ1 (ˆx)) ≥ 0. 1610 RAN ZHUO AND YAN LI

This is a contradiction. Hence (68) holds.

When Uλ0 ≡ 0, by the anti-symmetry of Uλ, that is,

λ0 Uλ0 (x) = −Uλ0 (x ), we derive that n Uλ0 (x) ≡ 0, x ∈ R . Obviously, α 2 (−∆) Uλ0 (x) = 0. Since

α 2 β1 λ0 τ1 λ0 β1 τ1 (−∆) Uλ0 (x) =u ¯ (x )¯v (x ) − u¯ (x)¯v (x) =u ¯β1 (xλ0 )(¯vτ1 (xλ0 ) − v¯τ1 (x)) = 0, it must be true that v¯(xλ0 ) =v ¯(x), x ∈ Rn, That is n Vλ0 (x) ≡ 0, x ∈ R .

Similarly, if Vλ0 (x) = 0 somewhere, then we can prove that n Uλ0 (x) ≡ 0, x ∈ R . When

λ0 Uλ0 (x),Vλ0 (x) > 0, x ∈ Σλ0 \{0 }, using an entirely similar argument of Step 2 in subcritical case, one can keep moving the plane Tλ. That is, there exists some small > 0, such that for any λ ∈ (λ0, λ0 + ), we have

λ Uλ(x) ≡ 0,Vλ(x) ≡ 0, ∀x ∈ Σλ \{0 }. (69)

This is a contradiction with the deﬁnition of λ0. Therefore (67) must not be true. We conclude that

n Uλ0 (x) = Vλ0 (x) ≡ 0, x ∈ R . This implies u and v are symmetric about some point in Rn.

Case (ii). λ0 = 0. In this case, we can move the plane from near x1 = +∞ to the left, and derive that

U0(x),V0(x) > 0, x ∈ Σ0. Hence

U0(x),V0(x) ≡ 0, x ∈ Σ0. This proves thatu ¯ andv ¯ are symmetric about the origin. So are u and v. In any case, u and v are symmetric about some point in Rn. This completes the proof of Theorem 1. NONEXISTENCE AND SYMMETRY OF SOLUTIONS 1611

REFERENCES [1] H. Berestycki and L. Nirenberg, On the method of moving planes and the sliding method, Bol. Soc. Brazil. Mat. (N.S.), 22 (1991), 1–37. [2] J. Bertoin, LévyProcesses, Cambridge Tracts in Mathematics, 121 Cambridge University Press, Cambridge, 1996. [3] J. P. Bouchard and A. Georges, Anomalous diffusion in disordered media, Statistical mechanics, models and physical applications, Physics reports, 195 (1990), 127–293. [4] L. Caffarelli and L. Vasseur, Drift diffusion equations with fractional diffusion and the quasi- geostrophic equation, Ann. Math., 171 (2010), 1903–1930. [5] L. Caffarelli, B. Gidas and J. Spruck, Asymptotic symmetry and local behavior of semilinear elliptic equations with critical Sobolev growth, Comm. Pure Appl. Math., 42 (1989), 271–297. [6] L. Caffarelli and L. Silvestre, An extension problem related to the fractional Laplacian, Comm. PDE., 32 (2007), 1245–1260. [7] L. Caffarelli and L. Silvestre, Regularity theory for fully nonlinear integro-differential equations, Comm. Pure Appl. Math., 62 (2009), 597–638. [8] M. Cai and L. Ma, Moving planes for nonlinear fractional Laplacian equation with negative powers, Disc. Cont. Dyn. Sys., 38 (2018), 4603–4615. [9] P. Constantin, Euler equations, navier-stokes equations and turbulence, Mathematical Foun- dation of Turbulent Viscous Flows Lecture Notes in Math., 1871 (2006), 1–43. [10] W. Chen and C. Li, Classification of solutions of some nonlinear elliptic equations, Duke Mathematical Journal, 63 (1991), 615–622. [11] W. Chen and C. Li, Methods on Nonlinear Elliptic Equations, AIMS book series on Diffreen- tial Equatons & Dynamical Systerms, Volume 4, 2010. [12] W. Chen and C. Li, A priori estimates for prescribing scalar curvature equations, Annals of Math., 145 (1997), 547–564. [13] W. Chen and C. Li, An integral system and the Lane-Emden conjecture, Disc. Cont. Dyn. Sys., 24 (2009), 1167–1184. [14] W. Chen, C. Li and Y. Li, A direct method of moving planes for the fractional Laplacian, Advances in Mathematics, 308 (2017), 404–437. [15] W. Chen, C. Li and B. Ou, Qualitative properties of solutions for an integral equation, Disc. Cont. Dyn. Sys., 12 (2005), 347–354. [16] W. Chen, C. Li and B. Ou, Classification of solutions for an integral equation, Comm. Pure Appl. Math., 59 (2006), 330–343. [17] C. Li and L. Ma, Uniqueness of positive bound states to Shrodinger systems with critical exponents, SIAM J. Math. Analysis, 40 (2008), 1049–1057. [18] T. Lin and J. Wei, Spikes in two coupled nonlinear Schrodinger equations, Ann. Inst. H. Poincare Anal. Non Linéaire, 22 (2005), 403–439. [19] B. Liu, Direct method of moving planes for logarithmic Laplacian system in bounded domains, Disc. Cont. Dyn. Sys., 38 (2018), 5339–5349. [20] L. Ma and L. Zhao, Sharp thresholds of blow up and global existence for the coupled nonlinear Schrödingersystem, J. Math. Phys., 49 (2008), 062103, 17 pp. [21] L. Silvestre, Regularity of the obstacle problem for a fractional power of the Laplace operator, Comm. Pure Appl. Math., 60 (2007), 67–112. [22] D. Tang and Y. Fang, Method of sub-super solutions for fractional elliptic equations, Dis. Con. Dyn. Sys., 23 (2018), 3153-3165. [23] V. Tarasov and G. Zaslasvky, Fractional dynamics of systems with long-range interaction, Comm. Nonl. Sci. Numer. Simul. 11 (2006), 885–889. [24] R. Zhuo and F. Li, Liouville type theorems for Schrödingersystems in a half space, Science China Mathematics, 58 (2015), 179–196. [25] R. Zhuo, F. Li and B. Lv, Liouville type theorems for Schrödingersystem with Navier boundary conditions in a half space, Comm. Pure Appl. Anal., 13 (2014), 977–990. [26] R. Zhuo, W. Chen, X. Cui and Z. Yuan, Symmetry and non-existence of solutions for a nonlinear system involving the fractional Laplacian, Dis. Con. Dyn. Sys., 36 (2016), 1125– 1141. Received January 2018; revised April 2018. E-mail address: [email protected] E-mail address: Yan [email protected]