IJNSNS 2021; aop Florinda Capone*, Maria Francesca Carfora, Roberta De Luca and Isabella Torcicollo Nonlinear stability and numerical simulations for a reaction–diffusion system modelling Allee effect on predators https://doi.org/10.1515/ijnsns-2020-0015 Received January 17, 2020; accepted May 31, 2021; published online August 5, 2021 Abstract: A reaction–diffusion system governing the prey–predator interaction with Allee effect onthe predators, already introduced by the authors in a previous work is reconsidered with the aim of showing destabilization mechanisms of the biologically meaning equilibrium and detecting some aspects for the eventual oscillatory pattern formation. Extensive numerical simulations, depicting such complex dynamics, are shown. In order to complete the stability analysis of the coexistence equilibrium, a nonlinear stability result is shown. Keywords: Alleeeffect;nonlinearstability;predator–prey;reaction–diffusion;Routh–Hurwitz;Turing–Hopf instabilities. 1 Introduction Ecological systems are characterized by the interaction between species and their natural environment. An important type of interaction which influences dynamics of all species is predation. Thus predator–prey models have been in the focus of ecological science since the early days of this discipline. It has turned out that predator–prey systems can show different dynamical behaviors (steady states, oscillations, chaos) depending on the value of model parameters. In predator–prey interactions, a crucial role is played by the cooperation which can help the survival of individuals. In fact, hunting cooperation, observed in many species (carnivores, birds, aquatic organisms, and spiders), implies, as a direct consequence, Allee effect in predators which allows these last to persist even if the prey population does not sustain them in the absence of hunting cooperation [1]. In many papers addressing population dynamics it is assumed that population is homogeneously distributed in the environment, leading to the formulation of ordinary differential equations (ODEs) systems [1–6]. If spatial domains are considered a spatial predator–prey system can be considered as a reaction–diffusion system. Reaction–diffusion systems have been studied in many fields as physics, *Corresponding author: Florinda Capone, Department of Mathematics and Applications ‘‘R.Caccioppoli’’, University of Naples Federico II, Via Cintia, Naples, Italy, E-mail: [email protected] Roberta De Luca, Department of Mathematics and Applications ‘‘R. Caccioppoli’’, University of Naples Federico II, Via Cintia, Naples, Italy, E-mail: [email protected] Maria Francesca Carfora and Isabella Torcicollo, Istituto per le Applicazioni del Calcolo ‘‘Mauro Picone’’, Via P. Castellino 111, CNR, Naples, Italy, E-mail: [email protected] (M.F. Carfora), [email protected] (I. Torcicollo) Open Access. © 2021 Florinda Capone et al., published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 2 | F. Capone et al.: On a reaction–diffusion system modelling Allee effect on predators chemistry, biology, economy and from many points of view ([7–13]). In particular it has been shown that reaction–diffusion systems are capable of self organized pattern formation. These spatial patterns arise not from inhomogeneity of initial or boundary conditions, but purely from the dynamics of the system, i.e. from the interaction of nonlinear reactions of growth processes and diffusion (as shown already by Turing14 [ ]). In particular the formation of stationary spatial patterns is due to Turing instabilities. The simultaneous appearance of Turing instability, which leads to steady spatial structures, with Hopf instability, which gives rise to temporal oscillations, is of great interest because these bifurcations are responsible for the breaking of spatial and temporal symmetries, respectively. The focus of research of many authors has been shifted to the study of the formation of such a type of spatio-temporal patterns. It has been shown that spatio-temporal patterns are very likely to be found in the neighborhood of Turing–Hopf bifurcations [15–17]. In this kind of bifurcation the formation of inhomogeneous stationary patterns caused by Turing instabilities “interacts” with the appearance of oscillations due to a Hopf bifurcation. In [7], the classical Lotka–Volterra model with logistic growth for the prey including hunting cooperation proposed in [1] has been extended by assuming that both prey and predators can linearly spread in the environment. Linear stability analysis has been performed and conditions guaranteeing that a biological meaningful equilibrium, stable in the absence of diffusion, becomes unstable in the presence of diffusion (Turing instability) have been found. For this model, we showed some numerical simulations which highlight the impact of different choices for the diffusion coefficients on the dynamics of the two populations and in particular we explored the Turing patterns formation. In the present paper, which can be seen as an enrichment of [7], we analyze again the problem with the aim of showing destabilization mechanisms of the biologically meaning equilibrium and detecting aspects for the eventual oscillatory pattern formation. Moreover, in order to complete the stability analysis of the coexistence equilibrium, we show a nonlinear stability result. The plan of the paper is as follows. Section 2 introduces the mathematical model while a nonlinear stability result is performed in Section 3.InSection 4 linear stability/instability results, both when populations are homogeneously mixed in the environment and in the spatial dependent case, are presented and extensive numerical simulations, depicting the theoretical aspects reported in the manuscript, are showed. 2 Mathematical model In this paper we reconsider the predator–prey model proposed in [7] concerning hunting cooperation with Allee effect on the predators. Assuming that both prey and predators can (linearly) diffuse in the environment, the model, in nondimensional form, that we investigate is given by ( ) ⎧ n n , ⎪ = n 1 − − (1 + p)np + 1Δn ⎨ t k (1) p ⎪ = (1 + p)np − p + Δp, ⎩ t 2 where the nondimensional variables are e x n = N, p = P, t = m, X = , (2) m m A and r eK̄ am d = , k = ,= ,= i , i = 1, 2. (3) m m 2 i A2m In particular, N(X,)andP(X,) denote, respectively, the prey and predator densities; for any ∈ {N, P},:(X,) ∈Ω×ℝ+ → (X,) ∈ ℝ+ being Ω a bounded, connected, and open subset of ℝn (n = 2, 3) having the internal cone propriety; A is the diameter of Ω. The (positive) constants appearing in (1)–(3) have the following biological meaning: r is the per capita intrinsic growth rate of prey; K̄ is the carrying capacity of prey; e is the food conversion efficiency; m is the per capita mortality rate of predators; is the attack rate; F. Capone et al.: On a reaction–diffusion system modelling Allee effect on predators | 3 a is the predator cooperation in hunting rate; and di are the diffusion coefficients. We append the following initial-boundary conditions { ∇n ⋅ n = 0, ∇p ⋅ n = 0, on Ω×ℝ+. , , , , . (4) n(x 0) = n0(x) p(x 0) = p0(x) on Ω being n the outward unit normal to Ω. , The biologically meaningful constant equilibria of system (1) are: (1) E0 = (0 0) (null solution); (2) , ∗ ∗, ∗ E1 = (k 0) (prey-only equilibrium); (3) E = (n p ) (coexistence equilibrium), with 1 n∗ = (5) 1 + p∗ and p∗ given by the positive solution of the equation [7] k2x3 + 2kx2 + k(1 − )x + (1 − k) = 0. (6) In order to perform the stability analysis for the coexistence equilibrium E∗ we set X = n − n∗, Y = p − p∗. (7) System (1) becomes ( ) ( )( ) ( ) ( ) X a11 a12 X 1ΔX F1 , = + + (8) t Y a21 a22 Y 2ΔY F2 with ⎧ 1 + 2p∗ a =− , a =− , ⎪ 11 ∗ 12 ∗ ⎪ k(1 + p ) 1 + p p∗ ⎪a = (1 + p∗)p∗, a = , ⎨ 21 22 1 + p∗ (9) ⎪ X2 F =− − (1 + p∗)XY − Y(XY + n∗Y + p∗X), ⎪ 1 k ⎪ ∗ ∗ ∗ . ⎩F2 = (1 + p )XY + Y(XY + n Y + p X) To (8) we associate the boundary conditions ∇X ⋅ n =∇Y ⋅ n = 0, on Ω×ℝ+ (10) andwedenotebyW∗(Ω) the functional space defined by { } , , d W∗(Ω)= ∈ W1 2(Ω) ∩ W1 2(Ω): =0onΩ×R+ . dn Let < < < < < 0 = 0 1 2 ··· n ··· (11) be the eigenvalues of the operator −Δ on Ω with homogeneous Neumann boundary conditions. The linear system linked to (8) is given by X = X, (12) t where ( ) ( ) a11 + 1Δ a12 , X . = X = (13) a21 a22 + 2Δ Y For each i = 0, 1, 2, 3, …, is an eigenvalue of if and only if is an eigenvalue of the matrix (see [18]) ( ) a − a ̃ 11 i 1 12 . (14) i = a21 a22 − i 2 4 | F. Capone et al.: On a reaction–diffusion system modelling Allee effect on predators ̃ The characteristic equation of i is 2 , i − I1i i + I2i = 0 (15) , , ̃ with Iji (j = 1 2), the principal i− invariants: ⎧ 0 , ⎪I1i = I1 − i( 1 + 2) ∗ (16) ⎨ 2 2 − k p 1 0, ⎪I2i = 1 2 i + i + I2 ⎩ k(1 + p∗) 0, , where I j (j = 1 2), are the principal invariants in absence of diffusion, i.e. kp∗ − ⎧I0 = a + a = , ⎪ 1 11 22 ∗ k(1 + p[) ] ⎨ (17) ⎪I0 = a a − a a = p∗ − + 1 + 2p∗ . ⎩ 2 11 22 12 21 k(1 + p∗)2 The necessary and sufficient conditions guaranteeing that all the roots of (15) have negative real part are the Routh–Hurwitz conditions [19] < , > , , , , . I1i 0 I2i 0 ∀i = 0 1 2 … (18) 3 Nonlinear stability Setting , , X = 1U1 Y = 2U2 (19) , , being i (i = 1 2) two positive scalings to be suitably chosen, (8) can be written as ⎧ U1 1 ̃ , ⎪ = b11U1 + b12U2 + 1ΔU1 + F1 t 1 ⎨ (20) ⎪ U2 1 ̃ , ⎩ = b21U1 + b22U2 + 2ΔU2 + F2 t 2 with j , ̃ ( , ). (21) bij = aij Fi = Fi 1U1 2U2 i In this section we perform the nonlinear stability analysis of the coexistence equilibrium with respect to the norm [20] [ 1 0 ‖ ‖2 ‖ ‖2 ‖ ‖2 V = I (‖U ‖ + ‖U ‖ ) + ‖b U − b U ‖ 2 2 1 2 11 2 21 1 ] ‖ ‖2 .