arXiv:0809.0665v1 [math.PR] 3 Sep 2008 ewe eegemaueadfiiemaue.I hspprthe paper this In measures intensity Poisson measures. for finite obtained and be measure can Lebesgue limits measure between of finite for kinds and what Lebesgue of for different quite are processes d neplt nsm a ewe h w rvoseteecss T parameters cases. the extreme of previous arrangements two fluctuatio different the time between way Occupation some question. in interpolate this investigating for used are d measure o-rva iis n t eainwt ..lcletnto ftesyst the of extinction local a.s. with relation one, its second and limits, non-trivial ntoseilcss eegemaue( measure Lebesgue cases: special two in measure mn h aaees eae eut o h orsodn ( de corresponding the regimes for dependence results long-range Related is different parameters. process the two limit among the has of part which temporal independ process the dimensions and low dimensions, In “low” dimensions. in behavior qualitative dependence different long-range separates which particular dimension critical a is it † 1 a (0 law symmetric a e words Key classifications: subject 2000 AMS given. rm tbepoes ogrnedependence. long-range process, stable orem, ∗ .I oecases some In ). h rtone, first The . etod netgc´nyd suisAazds ..1-4,M 14-740, A.P. Avanzados, Estudios de Investigaci´on y de Centro nttt fMteais nvriyo asw l aah ,02 2, Banacha ul. Warsaw, of University Mathematics, of Institute upre npr yCNCTGat464F(eio n ENGra MEiN and (Mexico) 45684-F Grant CONACyT by part in Supported h ( The β < oodmmweup [email protected] [email protected] [email protected] H µ ,α ,γ β, α, d, oazBojdecki Tomasz γ T ≤ ( d µ rnhn atcesse,sprrcs,ocpto iefluct time occupation superprocess, system, particle Branching : dx γ α [ = ) n trigfo nihmgnosPisnrno esr w measure random Poisson inhomogeneous an from starting and 1), cuaintm utainlmt o h ytmas system the for limits fluctuation time occupation , = ) sal ´v rcs (0 L´evy process -stable α d -rnhn atcesse ossso atce oigin moving particles of consists system particle )-branching H (2+ ihiiilihmgnosPisnstates Poisson inhomogeneous initial with dx/ = T cuaintmso rnhn systems branching of times Occupation ≡ α/β β 1+ (1 ) n nothers in and 1 − + γ 1 | n eae superprocesses related and ∨ x γ | eemnstene o ihdniyo o nodrt obtain to order in not or density high for need the determines , α γ ) )] γ , /β ≥ (if usG Gorostiza G. Luis .B en ftm rescaling time of means By 0. d < γ rmr 01,Scnay6J0 01,60G52. 60G18, 60J80, Secondary 60F17, Primary γ α < H ,α ,γ β, α, d, ,tehmgnoscs) n nt esrs( measures finite and case), homogeneous the 0, = T Abstract ∞ → ,itroae ewe h w xrm ae,and cases, extreme two the between interpolates ), ≤ 1 ) pitn ihaciia 1+ (1 critical a with splitting 2), as hr r w hehlsfrtedimension the for thresholds two are There . T ∞ → † xc 70 .. Mexico. D.F., 07000 exico 07Wrzw,Poland. Warszawa, -097 hg est ytm) h limit The systems). density (high .Teeoeteqeto arises question the Therefore s. ,α ,γ β, α, d, t10A12 (Poland). 1P03A01129 nt ∗ elmtpoessdpn on depend processes limit he naTalarczyk Anna T iisaeotie which obtained are limits n ftelmtpoess in processes, limit the of s edn nrelationships on pending ∞ → n nrmnsi “high” in increments ent measures T e efsmlrstable self-similar new a -uepoesaealso are )-superprocess mi icse.The discussed. is em htaeintermediate are that n oso intensity Poisson and aebe obtained been have ain ii the- limit uation, R d µ β γ codn to according 1 t intensity ith γ , )-branching .pl ∈ (0 > γ d , ], 1 Introduction
Occupation time fluctuation limits have been proved for the so-called (d,α,β)-branching particle systems in Rd with initial Poisson states in two special cases, namely, if the Poisson in- tensity measure is either Lebesgue measure, denoted by λ, or a finite measure [BGT1], [BGT2], [BGT3], [BGT4], [BGT6]. Those cases are quite special, as explained below, and the limit pro- cesses are very different. Therefore the question arises of what happens with Poisson intensity measures that are intermediate between Lebesgue measure and finite measures. That is the main motivation for the present paper, and our aim is to obtain limit processes that interpolate in some way between those of the two special cases. One of our objectives is to find out when the limits have long-range dependence behavior and to describe it. Another motivation is to derive analogous results for the corresponding superprocesses. In a (d,α,β)-branching particle system the particles move independently in Rd according to a standard spherically symmetric α-stable L´evy process, 0 < α ≤ 2, the particle lifetime is exponentially distributed with parameter V , and the branching law is critical with generating function 1 s + (1 − s)1+β, 0
d Nt(A) = number of particles in the Borel set A ⊂ R at time t. (1.2)
A common assumption for the initial distribution N0 is to take a Poisson random measure with locally finite intensity measure µ. The corresponding (d,α,β)-superprocess Y = (Yt)t≥0 is a measure-valued process, which is a high-density/short-life/small-particle limit of the particle system, with Y0 = µ. See [D], [E], [P2] for background on those particle systems and super- processes. In this paper we investigate (the limiting behavior of) the corresponding occupation time processes, i.e., t t Nsds, t ≥ 0, and Ysds, t ≥ 0. Z0 Z0 We recall that the distributions of these processes are characterized by their Laplace functionals as follows [GLM], [DP2]:
t d Eexp − hNs,ϕids = exp{−hµ, vϕ(t)i}, ϕ ∈ S(R ), (1.3) Z0 where vϕ(x, t) is the unique (mild) solution of the non-linear equation ∂ V v = ∆ v − v1+β + ϕ(1 − v ), (1.4) ∂t ϕ α ϕ 1+ β ϕ ϕ
vϕ(x, 0) = 0,
and t d Eexp − hYs,ϕids = exp{−hµ,uϕ(t)i}, ϕ ∈ S(R ), (1.5) Z0 2 where uϕ(x, t) is the unique (mild) solution of the non-linear equation ∂ V u = ∆ u − u1+β + ϕ, (1.6) ∂t ϕ α ϕ 1+ β ϕ
uϕ(x, 0) = 0, and ∆α is the infinitesimal generator of the α-stable process. (See the end of the Introduction for the standard notations h , i, S(Rd).) For µ = λ, N0 is homogeneous Poisson. This case is special (and technically simpler) because λ is invariant for the α-stable process (which implies in particular that ENt = λ for all t), and there is the following persistence/extinction dichotomy [GW], which heuristically explains the need for high density in some cases in order to obtain non-trivial occupation time fluctuation limits, and anticipates the situation we will encounter in this paper: (i) Persistence: If d > α/β, then Nt converges in law to an equilibrium state N∞ as t → ∞, such that EN∞ = λ. (ii) Extinction: If d ≤ α/β, then Nt becomes locally extinct in probability as t → ∞, i.e., for any bounded Borel set A, Nt(A) → 0 in probability. An analogous persistence/extinction dichotomy holds for the corresponding superprocess (with Y0 = λ) [DP1]. For α = 2 (Brownian motion) and d < 2/β a stronger extinction holds: the superprocess becomes locally extinct in finite time a.s. [I2], and we shall see that so does the particle system. The case of µ finite is special because the particle system goes to extinction globally in finite time a.s. for every dimension d, and so does the superprocess [P2]. The time-rescaled occupation time fluctuation process XT = (XT (t))t≥0 of the particle system is defined by 1 T t X (t)= (N − EN )ds, (1.7) T F s s T Z0 where Ns is given by (1.2) and FT is a norming. The problem is to find FT such that XT converges in distribution (in some way) as T →∞, and to identify the limit process and study its properties. This was done for µ = λ in the persistence case, d > α/β, [BGT3], [BGT4]. For the extinction case, d ≤ α/β, in [BGT6] we introduced high density, meaning that the initial Poisson intensity measure was taken of the form HT λ, with HT →∞ as T →∞, so as to counteract the local extinction, and we obtained similarly high-density limits with µ finite. For µ = λ and d > α/β the same results hold with or without high density (with different normings) [BGT6]. The limit processes are different for µ = λ and µ finite (some differences are mentioned below), and the results for any finite measure are essentially the same. In order to study asymptotics of XT as T → ∞ with measures µ that are intermediate between the two previous cases, we consider Poisson intensity measures of the form 1 µ (dx)= dx, γ ≥ 0. (1.8) γ 1+ |x|γ
We call the model so defined a (d,α,β,γ)- branching particle system, and a (d,α,β,γ)-superpro- cess the corresponding measure-valued process. To obtain non-trivial limits we multiply µγ by HT , which is suitably chosen in each case. Since µ0 = λ and µγ is finite for γ>d, by varying γ in the interval (0,d] we obtain limits that are between those of the two previous cases, which
3 are extreme in this sense, in a way that interpolates between them. The substantial role of γ was already noted in the simpler model of particle systems without branching [BGT5]. The above mentioned behaviors of Nt in the cases γ = 0 and γ >d raise the following questions on what happens for γ ∈ (0,d], and on its effect on asymptotics of XT : When does Nt suffer a.s. local extinction in the sense that for each bounded Borel set A there is a finite random time τA such that Nt(A) = 0 for all t ≥ τA a.s.? In this case the total occupation time ∞ 0 Nt(A)dt is finite a.s., and therefore high density is needed in order to obtain non-trivial limits for X . For γ > 0, N (A) converges to 0 in probability as t →∞ for any bounded Borel R T t set A and every dimension d, so local extinction in probability occurs, but the total occupation time may or may not be finite. It turns out that the threshold between the need for high density or not is given by d = α/β + γ, and then a natural question is whether d = α/β + γ is also the border to a.s. local extinction of the particle system. We will come back to this question. The limits for XT in [BGT6] are of three different kinds for both µ = λ and µ finite. In the first case there is a critical dimension, dc = α(1 + β)/β. For the “low” dimensions, d
4 of convergence in each case, nevertheless we expect that convergence in law in a space of continuous functions holds in all cases where the limit is continuous. We prove functional convergence only in the case of the above mentioned long-range dependence process ξ because of its special properties. A technical difficulty for the tightness proof is the lack of moments if β < 1. The time-rescaled occupation time fluctuation process for the (d,α,β,γ)-superprocess Y is defined analogously as (1.7), 1 T t X (t)= (Y − EY )ds, (1.10) T F s s T Z0 and the limits are obtained from (the proofs of) the results for the (d,α,β,γ)-branching particle systems, as a consequence of the fact that the log-Laplace equation of the occupation time of the superprocess is simpler than that of the particle system (see (1.3), (1.4), and (1.5), (1.6).) Our results on the fluctuation limits of superprocesses generalize those of Iscoe [I1], who considered the homogeneous case (γ = 0) only. Let us come back to the question of high density and local extinction for the (d,α,β,γ)- branching particle system. From Theorems 2.1, 2.5 and 2.6 it follows immediately (see Corol- lary 2.10) that in all cases where high density is not necessary (i.e., we may take HT ≡ 1) there is no a.s. local extinction (in spite of the fact that local extinction in probability occurs if γ > 0). For instance, condition (2.6) in Theorem 2.1(a) holds automatically if d > α/β + γ (with γ
5 d< 2/β, and he conjectured that an analogous result holds for α < 2 and d < α/β. So far as we know, this conjecture has not been proved. Summarizing, if γ
6 The transition probability density, the semigroup, and the potential operator of the standard d symmetric α-stable L´evy process on R are denoted respectively by pt(x), Tt (i.e., Ttϕ = pt ∗ ϕ) and (for d>α) ∞ ϕ(y) Gϕ(x)= Ttϕ(x)dt = Cα,d d−α dy, (1.12) R |x − y| Z0 Z where d−α Γ( 2 ) Cα,d = α d/2 α . (1.13) 2 π Γ( 2 )
Generic constants are written, C,Ci, with possible dependencies in parenthesis. Section 2 contains the results, and Section 3 the proofs.
2. Results
Given β ∈ (0, 1], let M be an independently scattered (1 + β)-stable measure on Rd+1 with control measure λd+1 (Lebesgue measure) and skewness intensity 1, i.e., for each Borel d+1 set A ⊂ R such that 0 < λd+1(A) < ∞, M(A) is a (1 + β)-stable random variable with characteristic function π exp −λ (A)|z|1+β 1 − i(sgn z)tan (1 + β) , z ∈ R, d+1 2 n o the values of M are independent on disjoint sets, and M is σ-additive a.s. (see [ST], Definition 3.3.1). For α ∈ (0, 2],γ ≥ 0, we define the process
1/(1+β) t −γ ξt = 11[0,t](r) pr(x − y)|y| dy pu−r(x)du M(drdx), t ≥ 0, (2.1) Rd+1 Rd Z Z Zr ! which is well defined provided that
t t 1+β −γ pr(x − y)|y| dy pu−r(x)du drdx < ∞, (2.2) Rd Rd Z Z0 Z Zr (see [ST]). For γ =0, ξ is the same as the process ξ defined by (2.1) in [BGT6]. We also recall the following process defined by (2.2) in [BGT6],
t 1/(1+β) ζt = 11[0,t](r)pr (x) pu−r(x)du M(drdx), t ≥ 0, (2.3) Rd+1 Z Zr which is well defined if d<α(2 + β)/(1 + β). We consider the (d,α,β,γ)-branching particle system described in the Introduction with XT defined by (1.7). Recall that the initial Poisson intensity measure is HT µγ. We formulate the results for low, critical and high dimensions separately, since, as mentioned in the Introduction, the qualitative behaviors of the limit processes are different in each one of these cases. In the theorems below K is a positive number depending on d,α,β,γ,V , which may vary from case to case and it is possible to compute it explicitly. The results for the low dimensions are contained in the following theorem.
7 Theorem 2.1 (a) Assume γ 2+ β γ ∨ α d<α − . (2.4) β β Then the process ξ given by (2.1) is well defined, and for 1+β 2+β−(dβ+γ)/α FT = HT T (2.5) with HT ≥ 1 and T 1−(d−γ)β/α lim =0, (2.6) T →∞ β HT we have XT ⇒C Kλξ. (b) Let γ ≥ d, 2+ β d<α , (2.7) 1+ β and put log T if γ = d, k(T )= (2.8) 1 if γ > d. Then for 1+β 2+β−(1+β)d/α FT = HT T k(T ) (2.9) with T lim =0, (2.10) T →∞ β β HT k(T ) we have XT ⇒f,i Kλζ, where ζ is defined by (2.3). Remark 2.2 (a) For γ = 0, Theorem 2.1(a) is the same as Theorem 2.2(a) in [BGT6]. For γ ≤ α and γ 8 [BGT3] we introduced a way of measuring long-range dependence in terms of the dependence exponent, defined by −θ κ = inf inf sup{θ> 0 : DT (z1, z2; u,v,s,t)= o(T ) as T → ∞}, (2.11) z1,z2∈R 0≤u DT (z1, z2; u,v,s,t) = | log Eei(z1(ξv−ξu)+z2(ξT +t−ξT +s) − log Eeiz1(ξv −ξu) − log Eeiz2(ξT +t−ξT +s)|. (2.12) (see also [RZ]). Proposition 2.3 Assume γ d if either α =2, or α< 2 and β > d−γ , κ = α d+α (2.13) d (1 + β − d−γ ) if α< 2 and β ≤ d−γ . α d+α d+α Remark 2.4 (a) Here, as in the case γ = 0 (Theorem 2.7 of [BGT3]), the intriguing phe- nomenon of two long-range dependence regimes occurs for α < 2. It seems also interesting to note that putting formally γ ≥ d in (2.13) we obtain κ = d/α (with no change of regime), which is indeed the dependence exponent of the process ζ (Proposition 2.9 of [BGT6]). On the other hand, the process ζ itself is not obtained from ξ by putting γ ≥ d. (b) If γ = 0 and β = 1, then ξ is the sub-fractional Brownian motion (multiplied by a constant) considered in [BGT], [BGT1]. We now turn to the critical dimensions, i.e., the cases where the inequalities in (2.4) and (2.7) are replaced by equalities. It turns out that in spite of different conditions on the normings, the limits have always the same form as for finite intensity measure, with the only exception of the case given in Theorem 2.5(a) below. Theorem 2.5 (a) Assume γ 9 (b) In all the remaining critical cases, i.e., (i) γ = α,γ 1+β 2 FT = HT (log T ) , (2.16) and (log T )1−β lim =0, (2.17) T →∞ β HT (ii) α<γ Theorem 2.6 (a) Assume γ EeihXt−Xs,ϕi π = exp − K(t1−γ/α − s1−γ/α) V |Gϕ(x)|1+β(1 − i(sgnGϕ(x)) tan (1 + β)) Rd 2 Z n d + 2cβϕ(x)Gϕ(x) dx , ϕ ∈ S(R ), t ≥ s ≥ 0, (2.25) o 10 X0 =0, where 0 if 0 <β< 1, c = (2.26) β 1 if β =1, and G is defined by (1.12). (b) Assume γ 1+β FT = HT log T, (2.27) ′ d and HT ≥ 1. Then XT ⇒f,i X, where X is an S (R )-valued process such that X0 =0, and for t> 0,Xt = X1 =(1+ β)-stable random variable determined by π EeihX1,ϕi = exp −K V |Gϕ(x)|1+β(1 − i(sgnGϕ(x)) tan (1 + β)) Rd 2 n Z d + 2cβϕ(x)Gϕ(x) dx , ϕ ∈ S(R ), (2.28) o with cβ given by (2.26). (c) Assume γ>α, 2+ β γ ∧ d d>α − , (2.29) β β 1+β FT = HT , (2.30) and −β lim THT =0. (2.31) T →∞ ′ d Then XT ⇒f,i X, where X is an S (R )-valued process such that X0 =0, and for t > 0,Xt = X1 = (1+ β)-stable random variable determined by π EeihX1,ϕi = exp −K V |Gϕ(x)|1+β(1 − i(sgnGϕ(x)) tan (1 + β)) Rd 2 Z d + 2cβϕ(x)Gϕ(x) Gµγ(dx) , ϕ ∈ S(R ), (2.32) o with cβ given by (2.26). Remark 2.7 (a) As in all the cases studied previously [BGT3], [BGT4], [BGT6], we observe the same phenomenon that in low dimensions the limit processes are continuous with a simple spatial structure and a complicated temporal structure (with long-range dependence), while in high dimensions they are truly S′(Rd)-valued with independent increments, and not necessarily continuous. (b) For low dimensions the forms of the limits depend on the relation between d and γ only, whereas for critical and high dimensions only the relationship between α and γ is relevant. More precisely, in critical dimensions we have different forms of the limits for γ<α and γ ≥ α, and in high dimensions the forms are different for γ <α,γ = α and γ>α. In the case γ>α even the normings are the same, depending only on β. (c) For β = 1 the limits are centered Gaussian. In high dimensions there is no continuous transition between the cases β < 1 and β = 1; in the latter case an additional term appears. 11 The coefficient cβ defined in (2.26) was introduced in order to present the results in unified forms. (d) We have assumed that the initial intensity measure is determined by µγ of the form (1.8). −γ It will be clear that the same results are obtained for the measure µγ(dx) = |x| dx if d>γ. Analogously as in the non-branching case [BGT5], other generalizations are also possible. Let us look further into the need for high density (i.e., to assume HT →∞) and the question of a.s. local extinction. For γ>d the Poisson intensity measure is finite, so there is only a finite number of particles at time t = 0 and the system becomes globally extinct in finite time a.s. due to the criticality of the branching. Also, for γ ∧ d>α it is not difficult to see that ∞ the total occupation time 0 Nsds is finite a.s. on bounded sets (see [BGT5], Proposition 2.1, because EN is the same for the systems with and without branching), so high density is also s R necessary. We have a more delicate situation in the remaining cases where the threshold is d = α/β + γ. Concerning extinction, the situation is completely clear for α = 2. In Theorem 2.8 below we state that for α = 2 and d < 2/β + γ there is a.s. local extinction, hence the total occupation time of any bounded set is finite a.s. We conjecture that the same is true for d < α/β + γ if α< 2, but we have not been able to prove it. Theorem 2.8 Assume α =2. If d< 2/β + γ, then for each bounded Borel set A, P [there exists τA < ∞ such that Nt(A)=0 for all t ≥ τA]=1. The proof of this theorem relies on Iscoe’s a.s. local extinction result for the superprocess [I2], by showing that in general (i.e., for 0 <α ≤ 2) a.s. local finiteness of the total occupation time of the (d,α,β,γ)-superprocess implies a.s. local extinction of the (d,α,β,γ)-branching particle system. On the other hand, as explained in the Introduction, for α < 2 the a.s. local extinction for the superprocess cannot occur, and we do not know how to prove directly the a.s. local finiteness of its total occupation time. The next ergodic-type result, which is a direct generalization of [Ta], shows that α/β + γ is indeed a natural threshold. Proposition 2.9 Assume α γ<α, d = + γ, F = T 1−γ/α, (2.33) β T and denote 1 T t Z (t)= N ds, t ≥ 0. (2.34) T F s T Z0 Then ZT ⇒C λξ, where ξ is a real non-negative process with Laplace transform −γ Eexp{−θ1ξt1 −···− θnξtn } = exp − v(x, τ)|x| dx , (2.35) Rd Z for any τ > 0, where θ1,...,θn ≥ 0, 0 ≤ t1 < ··· < tn ≤ τ, and v(x, t) is the unique non- negative solution of the equation t V t v(x, t)= p (x)ψ(τ − s)ds − T v1+β(·,s)(x)ds, (2.36) t−s 1+ β t−s Z0 Z0 12 with n ψ(s)= θk11[0,tk](s). (2.37) Xk=1 To complete the discussion on a.s. local extinction we formulate a corollary which follows immediately from our results but which, nevertheless, seems worth stating explicitly. Corollary 2.10 If γ<α and d ≥ α/β + γ, then the (d,α,β,γ)-branching particle system does not have the a.s. local extinction property. Indeed, for d > α/β + γ, from Theorems 2.1, 2.5 and 2.6 it follows that one may take T HT ≡ 1. By Proposition 2.1 of [BGT5], E 0 hNs,ϕids (for Rd ϕ(x)dx =6 0) is of larger order than F as T →∞ hence, by (1.7), for any bounded Borel set A, ∞ N (A)ds = ∞ a.s., which T R R 0 s excludes a.s. local extinction. For d = α/β + γ the result follows immediately from Proposition R 2.9. We end with the results for the superprocess. Theorem 2.11 Let Y be the (d,α,β,γ)-superprocess and XT its occupation time fluctuation process defined by (1.10). Then the limit results for XT as T → ∞ are the same as those in Theorems 2.1, 2.5 and 2.6, with the same normings, and cβ =0 in all cases in Theorem 2.6. 3. Proofs 3.1 Scheme of proofs The proofs of Theorems 2.1, 2.5 and 2.6 follow the general scheme presented in [BGT6]. For completeness we recall the main steps. As explained in [BGT3, BGT4, BGT6], in order to prove convergence ⇒i it suffices to show e e lim Ee−hXT ,Φi = Ee−hX,Φi (3.1) T →∞ d+1 for each Φ ∈ S(R ), Φ ≥ 0, where X is the corresponding limit process and XT , X are defined by (1.11). To prove convergence ⇒C according to the space-time approach [BGR] it is enough d to show additionally that the family {hXT ,ϕi}T ≥1 is tight in C((0, τ], R),ϕe∈ Se(R ),τ > 0. Without loss of generality we may fix τ = 1 (see (1.11)). To simplify slightly the calculations we consider Φ of the form Φ(x, t)= ϕ ⊗ ψ(x, t)= ϕ(x)ψ(t), ϕ ∈ S(Rd), ψ ∈ S(R),ϕ,ψ ≥ 0. Denote 1 1 t ϕ = ϕ, χ(t)= ψ(s)ds, χ (t)= χ . (3.2) T F T T T Zt We define t x vT (x, t)=1 − Eexp − hNr ,ϕT iχT (T − t + r)dr , 0 ≤ t ≤ T, (3.3) Z0 13 where N x is the empirical process of the branching system started from a single particle at x. The following equation for vT was derived in [BGT3] (formula (3.8), see also [BGT1]) by means of the Feynman-Kac formula: t V v (x, t)= T ϕ χ (T − u)(1 − v (·,u)) − v1+β(·,u) (x)du, 0 ≤ t ≤ T. (3.4) T t−u T T T 1+ β T Z0 This equation (with T = 1) is the space-time version of equation (1.4). It is the log-Laplace t equation for L (as in (1.11)), where L is the occupation time Lt = 0 Nsds. Formulas (3.3) and (3.4) imply t R e 0 ≤ vT ≤ 1, vT (x, t) ≤ Tt−uϕT (x)χT (T − u)du. (3.5) Z0 For brevity we denote H ν (dx)= H µ (dx)= T dx. (3.6) T T γ 1+ |x|γ By the Poisson property and (3.4) we have T e −hXT ,ϕ⊗ψi Ee = exp − vT (x, T )νT (dx)+ TuϕT (x)χT (u)duνT (dx) (3.7) Rd Rd Z Z Z0 V V = exp I (T )+ I (T ) − I (T ) , (3.8) 1+ β 1 2 1+ β 3 where T s 1+β I1(T ) = TT −s Ts−uϕT χT (T − u)du (x)dsνT (dx), (3.9) Rd Z Z0 "Z0 # T I2(T ) = TT −s(ϕT χT (T − s)vT (·,s))(x)dsνT (dx), (3.10) Rd Z Z0 T s 1+β 1+β I3(T ) = TT −s Ts−uϕT χT (T − u)du − vT (·,s) (x)dsνT (dx). (3.11) Rd Z Z0 "Z0 # In the proofs of Theorems 2.1, 2.5 and in Theorem 2.6 for β < 1 we show V −hX,ϕe ⊗ψi lim exp I1(T ) = Ee , (3.12) T →∞ 1+ β and lim I2(T )=0, (3.13) T →∞ where (3.13) is obtained from C T T I2(T ) ≤ 2 Ts(ϕTuϕ)(x)dudsνT (dx) (3.14) F Rd T Z Z0 Z0 (see (3.5)). In Theorem 2.6 for β = 1 the limit of I2(T ) is non-trivial and corresponds to the expressions involving cβ (see (2.25), (2.26), (2.28),(2.32)). In all the cases lim I3(T ) = 0 (3.15) T →∞ 14 By the argument in [BGT6], in order to prove (3.15) we show lim J1(T ) = 0 (3.16) T →∞ and lim J2(T )=0, (3.17) T →∞ where T s u 1+β J1(T ) = TT −s Ts−u ϕT Tu−rϕT dr du (x)dsνT (dx) Rd Z Z0 "Z0 Z0 # 1+β 1 T T T ≤ 2+2β Ts Tu ϕ Trϕdr du (x)dsνT (dx), (3.18) F Rd 0 0 0 T Z Z "Z Z # 1+β T s u 1+β J2(T ) = TT −s Ts−u Tu−rϕT dr du (x)dsνT (dx) Rd Z Z0 Z0 Z0 ! 1+β 1+β 1 T T T ≤ (1+β)(1+β) Ts Tu Trϕdr du (x)dsνT (dx).(3.19) Rd FT Z Z0 Z0 Z0 ! We remark that the proof of (3.15) is the only place where the high density (with specific conditions on HT ) is required in some cases. Finally, the ⇒f convergence is obtained by an analogous argument as explained in the proof of Theorem 2.1 in [BGT4]. 3.2 Auxiliary estimates Recall that the transition density pt of the standard α-stable process has the self-similarity property −d/α −1/α d pat(x)= a pt(a x), x ∈ R , a> 0, (3.20) and it satisfies C C 1 ≤ p (x) ≤ 2 , (3.21) 1+ |x|d+α 1 1+ |x|d+α where the lower bound holds for α< 2. Denote 1 f(x)= ps(x)ds. (3.22) Z0 The following estimate can be easily deduced from (3.20) and (3.21): C f(x) ≤ , (3.23) |x|d+α C if d < α, f(x) ≤ C(1 ∨ log |x|−1) if d = α, (3.24) d−α C/|x| if d > α. 15 We will also use the following elementary estimates: Let 0 γ sup |y| fγ (y) < ∞, (3.30) |y|>1 and C if γ < α, −1 fγ(y) ≤ C(1 ∨ log |y| ) if γ = α, (3.31) γ−α C/|y| if γ > α. 3.3 Proof of Theorem 2.1(a) According to the scheme sketched above, in order to prove ⇒f,i convergence we show (3.1). By (3.7)-(3.11)and (2.1) it is enough to prove (3.12), which amounts to 1 1 1+β dx 1+β lim I1(T )= ps(x − y) pu−s(y)χ(u)du dyds γ ϕ(z)dz , T →∞ Rd 0 Rd s |x| Rd Z Z Z Z Z (3.32) (see (3.9)) and, additionally, (3.13) and (3.15). To simplify the notation we will carry out the −γ proof for µγ of the form µγ(dx)= |x| dx instead of (1.8). It will be clear that in the present case (d<γ) this will not affect the result. ′ ′ By (3.9), (3.2), (3.6), the definition of Tt, substituting s =1 − s/T,u =1 − u/T , we have I1(T ) 2+β 1 1 1+β T HT −γ = 1+β pTs(x − y) pT (u−s)(y − z)ϕ(z)χ(u)dzdu |x| dydsdx. F Rd 0 Rd s Rd T Z Z Z Z Z (3.33) 16 Denote d/α 1/α ϕT (x)= T ϕ(T x) (3.34) and e 1 gs(x)= pu−s(x)χ(u)du, 0 ≤ s ≤ 1. (3.35) Zs Observe that gs ≤ Cf, (3.36) where f is defined by (3.22). By (3.20) and (2.5), making obvious spatial substitutions in (3.33), we obtain 1 1+β −γ I1(T )= ps(x − y)(gs ∗ ϕT (y)) |x| dxdsdy. (3.37) Rd Rd Z Z0 Z −γ Note that if we consider the measure µγ of the forme (1.8), then in (3.37) instead of |x| −γ/α γ −1 1 d we have (T + |x| ) . Since gs ∈ L (R ), by (3.34) it is clear that gs ∗ ϕT (y) converges to gs(y) Rd ϕ(z)dz almost everywhere in y. Hence, to prove (3.32) it remains to justify the passage to the limit under the integrals in (3.37). Denote R e 1 1+β −γ hT (y)= ps(x − y)(gs ∗ ϕT (y)) |x| dxds. (3.38) Rd Z0 Z First we prove pointwise convergence of hT , which amountse to showing that the integrand is majorized by an integrable function independent of T . Fix y =6 0. We use (3.36) and observe that (T 1/α|x|)dϕ(T d/αx) f ∗ ϕ (y) = f(y − z)ϕ (z)dz + f(y − x) dx T T |x|d Z|z|≤|y|/2 Z|z|>|y|/2 y C e ≤ f ϕ e(z)dz + f(y − x)dz 2 T |y|d Z|z|≤|y|/2 Z|z|>|y|/2 y C e ≤ f ϕ(z)dz + d , (3.39) 2 Rd |y| Z by the unimodal property of the α-stable density and since ϕ ∈ S(Rd). We conclude by noting that 1 −γ ps(x − y)|x| dsdx < ∞ for y =06 , Rd Z Z0 by (3.31). Since (see (3.29)) 1+β hT (y) ≤ fγ (y)((f ∗ ϕT )(y)) , (3.40) to prove convergence of I1(T ) it suffices to show that the right-hand side of (3.40) (denoted by ∗ 1 Rd e hT ) converges in L ( ) as T →∞. If γ<α, then fγ is bounded by (3.31), and the assumption (2.4) implies that f ∈ L1+β(Rd), (3.41) 17 1+β 1 d so (f ∗ ϕT ) converges in L (R ). ∗ 1 Rd Next assume γ ≥ α. It is easily seen that hT 11{|y|≥1} converges in L ( ), by (3.41) and ∗ 1 Rd (3.30).e To prove that hT (y)11{|y|<1} converges in L ( ) too, it suffices to find p,q > 1, 1/p + 1/q = 1, such that p d fγ(y)11{|y|<1} ∈ L (R ) (3.42) and f 1+β ∈ Lq(Rd). (3.43) If γ = α, then (3.31) implies that (3.42) holds for any p > 1, and by (2.4) it is clear that (3.43) is satisfied for q sufficiently close to 1. If γ>α, condition (2.4) is equivalent to γ − α (1 + β)(d − α) + < 1, d d so we can take p and q such that 1/p +1/q = 1, 1 γ − α 1 (1 + β)(d − α) > and > . p d q d For such p and q we have (3.42) and (3.43) by (3.31), (3.23) and (3.24). This completes the proof of (3.32). We proceed to the proof of (3.13). By (3.14), applying the same substitutions as for I1(T ) and using the notation (3.34) we have 2−d/α−γ/α HT T I2(T ) ≤ C 2 fγ(y)ϕT (y)(f ∗ ϕT )(y)dy. (3.44) F Rd T Z Assume γ<α. By (3.31), e e 2−d/α−γ/α HT T I2(T ) ≤ C1 2 ||ϕT (f ∗ ϕT )||1 FT 2−d/α−γ/α HT T ≤ C1 ||ϕe||1||f||e1+β||ϕT || 1+β → 0 2 β FT by (2.4), and since e (d/α)(p−1)/p ||ϕT ||p = ||ϕ||pT for p ≥ 1. (3.45) Next, let γ ≥ α. By (3.44) and (3.30) we have e ′ ′′ I2(T ) ≤ C2(I2(T )+ I2 (T )), (3.46) where 2−d/α−γ/α ′ HT T I2(T )= 2 ||ϕT (f ∗ ϕT )||1, (3.47) FT and 2−d/α−γ/α e e ′′ H T I (T )= T f (y)ϕ (y)(f ∗ ϕ )(y)dy. (3.48) 2 F 2 γ T T T Z|y|≤1 18 e e By the H¨older inequality, 2−d/α−γ/α ′ HT T I2(T ) ≤ 2 ||ϕT ||p||f||q||ϕ||1 (3.49) FT for any p, q ≥ 1, 1/p +1/q =1. If 1/q > (d − α)/d,e then ||f||q < ∞, and if 1/q is sufficiently close to (d − α)/d, then by (3.45), (2.5) and (2.4) the right-hand side of (3.49) converges to 0 as T →∞. ′′ We estimate I2 (T ) using the generalized H¨older inequality 2−d/α−γ/α ′′ HT T I2 (T ) ≤ 2 ||fγ11{|·|≤1}||r||ϕT ||p||f||q||ϕ||1 FT for r, p, q ≥ 1, 1/p +1/r +1/q = 1. We take r, q such thate 1/r > (γ − α)/d (then the r-norm will be finite by (3.31)) and 1/q > (d − α)/d. By (2.4), it is easily seen that one can choose r, q as above and p = α/(2α − γ)+ ε for ε> 0 sufficiently small (note that by (2.4), 2α>γ). ′′ Then by (3.45) and (2.5) we obtain that I2 (T ) → 0 as T →∞. Thus, we have proved (3.13). According to the general scheme, in order to obtain (3.15) it suffices to show (3.16) and (3.17). The proofs are quite similar to the argument presented above, therefore we omit the proof of (3.16) and we give an outline of the proof of (3.17), since this is the only place where the condition (2.6) is needed. By (3.19), (2.5) and the usual substitutions we have T 1−(d/α)β+(γ/α)β J2(T ) ≤ C β R(T ), (3.50) HT where 1+β 1+β R(T )= fγ (y)(f ∗ (f ∗ ϕT ) ) (y)dy. (3.51) Rd Z By (2.6), to prove (3.17) it remains to show that e sup R(T ) < ∞. (3.52) T If γ<α then, by (3.31), 1+β 1+β 1+β (1+β)(1+β) (1+β)(1+β) R(T ) ≤ C1||f ∗ (f ∗ ϕT ) ||1+β ≤ C1||f||1+β||f||1+β ||ϕ||1 < ∞, (3.53) by (3.41). e If γ ≥ α, then we write R(T )= ... + .... Z|y|>1 Z|y|≤1 By (3.30) the first integral can be estimated as in (3.53), and the second one is bounded by 1+β (1+β)(1+β) (1+β)(1+β) ||fγ11{|·|≤1}||p||f||q(1+β)||f||1+β ||ϕ||1 , where 1/p +1/q = 1. We already know that there exist such p and q that this expression is finite (see (3.42) and (3.43)). 19 We have thus established the convergence XT ⇒f,i Kλξ. In order to obtain ⇒C convergence it suffices to show that the family {hXT ,ϕi}T ≥1 is tight in C([0, 1], R) for any ϕ ∈ S(Rd) ([Mi]). One may additionally assume that ϕ ≥ 0. We apply the method presented in [BGT3] and [BGT6]. We start with the inequality 1/δ P (|hXT ,ϕ ⊗ ψi| ≥ δ) ≤ Cδ (1 − Re(E exp{−iθhXT ,ϕ ⊗ ψi}))dθ, (3.54) Z0 R valid for any ψ ∈e S( ),δ > 0. Fix 0 ≤ t1 < t2 ≤ 1 and take ψ eapproximating δt2 − δt1 such 1 that χ(t)= t ψ(s)ds satisfies 0 ≤ χ ≤ 11 . (3.55) R [t1,t2] Then the left hand side of (3.54) approximates P (|hXT (t2),ϕi−hXT (t1),ϕi| ≥ δ). So, in order to show tightness one should prove that the right hand side of (3.54) is estimated by h h 1+σ C(t2 − t1 ) for some h,σ > 0. By the argument in [BGT6] this reduces to showing that h h 1+σ A(T ) ≤ C(t2 − t1 ) (3.56) and h h 1+σ I1(T ) ≤ C(t2 − t1 ) , (3.57) where I1 is defined by (3.9), and T s HT s u −γ A(T )= 2 TT −s(ϕTs−uϕ)(x)χ 1 − χ 1 − |x| dudsdx. (3.58) FT Rd 0 0 T T Z Z Z The proofs of (3.56) and (3.57) are quite involved and lengthy, therefore, as an example we present only the argument for the case γ<α, which, together with (2.4) implies 1+ β d<α . (3.59) β We start with (3.57). By self-similarity of ps we have −γ −γ/α d ps(x − y)|x| dx ≤ Cs , y ∈ R . (3.60) Rd Z Using this, (3.37), (3.35) and (3.55) we obtain I1(T ) ≤ C(W1(T )+ W2(T )), (3.61) 20 where t1 t2 1+β −γ/α W1(T ) = s pu−s(y − z)ϕT (z)dzdu dsdy, (3.62) Rd Rd Z Z0 Zt1 Z t2 t2 1+β −γ/α e W2(T ) = s pu−s(y − z)ϕT (z)dzdu dsdy. (3.63) Rd Rd Z Zt1 Zs Z Fix any ρ such that e d 1 max − , 0 <ρ< 1 (3.64) α β (see (3.59). For any fixed s ∈ [0, t1] we apply the Jensen inequality to the measure (u − s)−ρ 11[t ,t ](u)du, t2 (r − s)−ρdr 1 2 t1 obtaining R t1 t2 β t2 −γ/α −ρ −ρ ρ 1+β W1(T ) ≤ s (r − s) dr (u − s) ((u − s) pu−s ∗ ϕT (y)) dudsdy. Rd Z Z0 Zt1 Zt1 We have e −(d/α)β/(1+β) ||pu−s ∗ ϕT ||1+β ≤ ||pu−s||1+β||ϕ||1 =(u − s) ||p1||1+β||ϕ||1, (3.65) hence e t2 t1 (1−ρ)β −γ/α ρβ−(d/α)β W1(T ) ≤ C(t2 − t1) s (u − s) dsdu, Zt1 Z0 which, after the substitution s′ = s/u, by (3.64) and γ<α, yields (1−ρ)β h h h h 1+(1−ρ)β W1(T ) ≤ C1(t2 − t1) (t2 − t1 ) ≤ C2(t2 − t1 ) , (3.66) where h =2 − γ/α + ρβ − (d/α)β > 0 by assumptions. Next, by (3.63) we have t2 t2−t1 1+β −γ/α W2(T ) ≤ s pudu ∗ ϕT (y) dsdy. Rd Z Zt1 Z0 ′ The Young inequality, substitution u = u/(t2 − t1), and self-similaritye imply 1−γ/α 1−γ/α 1+β−(d/α)β 1+β 1+β W2(T ) ≤ C(t2 − t )(t2 − t1) ||f||1+β||ϕ||1 1−γ/α 1−γ/α 2+β−(d/α)β ≤ C1(t2 − t1 ) , (3.67) by (3.41). Combining (3.66), (3.67), (3.61) and using (3.59), we obtain (3.57). It remains to prove (3.56). 21 Applying the usual substitutions to A(T ) given by (3.58) we obtain A(T ) 2−d/α−γ/α 1 1 HT T −γ = 2 |x| ps(x − y)ϕT (y)pu−s(y − z)ϕT (z)χ(s)χ(u)dxdzdyduds, F R3d T Z0 Zs Z hence, by (3.60), (3.55) and the H¨older inequality,e e 2−d/α−γ/α t2 t2 HT T −γ/α A(T ) ≤ C 2 s ||ϕT || 1+β ||pu−s ∗ ϕT ||1+βduds. F β T Zt1 Zs We use (3.65), (3.45) and (2.5), obtaining e e 1−γ/α 1−1/α 1−(d/α)β/(1+β) A(T ) ≤ C1(t2 − t1 )(t2 − t1) , which implies (3.56) by (3.59). This completes the proof of tightness. ✷ 3.4 Proof of Theorem 2.1(b) We prove the theorem for 2+ β γ = d< α (3.68) 1+ β (see Remark 2.2(c)). Recall that in this case k(T ) occurring in (2.9) and (2.10) is log T . According to the discussion in Section 3.1 it suffices to prove (3.12), (3.13) and (3.15). By the form of the limit process (see (2.3)), (3.12) is equivalent to 1 1 1+β 1+β σ(Sd−1) lim I1(T )= ps(y) pu−s(y)χ(u)du dsdy ϕ(z)dz , (3.69) T →∞ α Rd 0 s Rd Z Z Z Z d where σ(Sd−1) is the measure of the unit sphere in R (= 2 if d = 1). By (3.9), (3.2), (3.6), using similar substitutions as in the previous section, we obtain I1(T ) 1 1 1+β 1 −1/α 1 = ps(xT − y) pu−s(y − z)χ(u)ϕT (z)dzdu d dydsdx, log T Rd Rd Rd 1+ |x| Z Z0 Z Zs Z (3.70) e where ϕT is given by (3.34). We write ′ ′′ ′′′ I1(T )= I (T )+ I (T )+ I (T ), (3.71) e 1 1 1 where 1 ′ 1 I1(T ) = ... (3.72) log T 1/α Rd Z1<|x| T 1/α 1 d−1 ′ 1 −1/α 1+β r I1(T )= ps(wrT − y)(gs ∗ ϕT (y)) d dydsσ(dw)dr, log T Rd 1+ r Z1 ZSd−1 Z0 Z where g is defined by (3.35). The crucial step is the substitutione log r r′ = , (3.75) log T which gives 1/α 1 rd ′ r−1/α 1+β T I1(T )= ps(wT − y)(gs ∗ ϕT (y)) rd dydsσ(dw)dr. Rd 1+ T Z0 ZSd−1 Z0 Z ′ It is now clear that if one could pass to the limit undere the integrals as T → ∞, then I1(T ) would converge to the right hand side of (3.69). This procedure is indeed justified by the fact that for f defined by (3.22) we have f ∈ L2+β(Rd), (3.76) which follows from (3.68), (3.23) and (3.24). We omit the details, which are similar to the argument in [BGT6] (see (3.51) therein). ′′ ′′′ ′′ Next we show that I1 (T ) and I1 (T ) tend to zero. In I1 (T ) (see (3.73)) we substitute x′ = xT −1/α and we use (3.36), obtaining 1 d/α ′′ C 1+β T I1 (T ) ≤ ps(x − y)(f ∗ ϕT (y)) d d/α dydsdx log T Rd 1+ |x| T Z|x|>1 Z0 Z C C ≤ 1 ||f ∗ ϕ ||1+β ≤ 1 ||f||1+eβ||ϕ||1+β → 0. log T T 1+β log T 1+β 1 ′′′ I1 (T ) (see (3.74)) is estimatede as follows: ′′′ C I (T ) ≤ f ∗ (f ∗ ϕ )1+β(xT −1/α)dx 1 log T T Z|x|≤1 C ≤ 2 ||f|| ||(f ∗ ϕe)1+β|| log T 2+β T (2+β)/(1+β) C2 2+β 1+β ≤ ||f|| ||ϕ|| e → 0, log T 2+β 1 by (3.76). This and (3.71) prove (3.69). To prove (3.13) we use (3.14) and easily obtain 2−2d/α CHT T −1/α 1 I2(T ) ≤ 2 f ∗ (ϕT (f ∗ ϕT ))(xT ) d dx. F R2 1+ |x| T Z We write the right-hand side as the sum of integralse e over {|x| ≤ T 1/α} and {|x| > T 1/α}. To estimate the integral over {|x| ≤ T 1/α} we use 1 1 sup d dx < ∞, (3.77) log T 1/α 1+ |x| T>2 Z|x|≤T 23 and in the second integral we apply 1/(1 + |x|d) ≤ T −d/α. For each of the integrals we use appropriately the H¨older inequality, properties of the convolution and (3.45), obtaining the es- 2((d/α)1/(2+β)−1/(1+β)) (d/α)1/(2+β)−2/(1+β) timates C1T and C2T , respectively (the factors involving negative powers of HT and log T have been estimated by constants). These bounds tend to zero as T →∞ by (3.68). We omit details. This proves (3.13). To prove (3.16) and (3.17) we use (3.18) and (3.19). Again, we consider separately the 1/α 1/α integrals over {|x| ≤ T } and {|x| > T }, and apply the same tricks as for I2(T ). For J1(T ) we obtain the estimate (d/α)(1+β)/(2+β)−1 J1(T ) ≤ CT → 0 (log T and HT appear with negative powers only), whereas T T J2(T ) ≤ C1 + C2 → 0 β β β 1+β HT (log T ) HT (log T ) by assumption (2.10). The proof of Theorem 2.1 is complete ✷ 3.5 Proof of Proposition 2.3 Properties (a)-(c) are clear, following from (2.1) and Theorem 2.1(a). Recall that the index R of self-similarity is defined as a ∈ such that the process (ξct)t∈R+ has the same distribution a as (c ξt)t∈R+ for any c> 0. To calculate the dependence exponent of ξ (see (2.11), (2.12)) first note that by (2.1) and Proposition 3.4.2 of [ST] the finite-dimensional distributions of ξ are given by Eexp{i(z1ξt1 + ··· + zkξtk )} k 1/(1+β) tj 1+β −γ = exp − zj pr(x − y)|y| dy 11[0,tj ](r) pu−r(x)du Rd+1 Rd " j=1 r Z X Z Z k 1/(1+β) −γ × 1 − i sgn zj pr(x − y)|y| dy Rd j=1 X Z tj π × 11 (r) p (x)du tan (1 + β) drdx . (3.78) [0,tj ] u−r 2 Zr The argument goes along the lines of the proof of Theorem 2.7 of [BGT3]. For fixed z > 0 + − and 0 ≤ u CT −d/α if either α =2 or β > (d − γ)/(d + α), D± ≤ T CT −(d/α)δ for any β<δ< 1+ β +(d − γ)/(d + α) if α< 2, β ≤ (d − γ)/(d + α), and for T sufficiently large, + −d/α DT ≥ CT , + −(d/α)δ DT ≥ CT for any δ > 1+ β − (d − γ)/(d + α) if α< 2, β ≤ (d − γ)/(d + α). (3.79) 24 The upper estimates are obtained similarly as (4.3), (4.4) in [BGT3] and (3.108) in [BGT6]. The only difference is that in formulas (4.9) and (4.10) in [BGT3] a new factor, Rd pr(x − y)|y|−γdy, appears (which corresponds to p (x) in (3.101) in [BGT6]). This factor is responsible r R for the new long-range dependence threshold and the form of the dependence exponent (2.13). In the estimates we use (3.30). The first of the lower estimates is obtained exactly as (4.18) in [BGT3]. The new expression, (u+v)/2 −γ pr(x − y)|y| dydxdr, Rd Zu Z|x|≤1 Z that appears at the right-hand side is finite by (3.31). To derive (3.79) we argue as in (4.22), (4.24) of [BGT3] and we apply estimates (4.21) (which holds for |x| ≤ T 1/α) and (4.23) therein, obtaining (u+v)/2 + −(d/α)(1+β)+εβ(d+α) −γ DT ≥ CT pr(x − y)|y| dydxdr, (3.80) d/(d+α)α−ε Rd Zu+(v−u)/4 Z1≤|x|≤T Z where ε> 0 is sufficiently small. For 1 ≤|x| ≤ T d/(d+α)α−ε we have −γ −γ pr(x − y)|y| dy ≥ C|x| pr(x − y)dy Rd Z Z|x−y|≤1/2 −γ −dγ/d(d+α)α+εγ ≥ C1|x| inf inf pr(z) ≥ C2T . v−u u+v 1 4 +u Putting this into (3.80) we obtain (3.79). ✷ 3.6 Proof of Theorem 2.5 Each of the cases requires a different proof, and none of them is straightforward. We will present a detailed proof of the part (a) only. In the remaining cases we will confine ourselves to explaining why the limit processes have the forms given in the theorem (recall that part (b)(iv) has been proved in [BGT6]). It seems instructive to compare the proofs for this theorem to the argument given in the proof of Theorem 2.1(b) for γ = d. Although the critical cases are of different kinds, some of the technical tricks repeat in all cases, nevertheless they are applied in a slightly different way and are far from being identical. Proof of case (a) To simplify calculations we again consider the measure µγ of the form −γ µγ(dx)= |x| dx instead of (1.8). In (3.9) we substitute u′ = s − u and then s′ =(T − s)/T , obtaining 1+β 1 T (1−s) HT T u −γ I1(T )= 1+β psT (x−y) pu(y − z)ϕ(z)χ s + dzdu |x| dydsdx. F Rd 0 Rd 0 Rd T ! T Z Z Z Z Z Using (2.15), (3.20) and substitution x′ = x(sT )−1/α we have 25 1 1 −1/α I1(T ) = p1(x − y(sT ) ) log T Rd 0 Rd Z Z Z 1+β T (1−s) u −γ/α −γ × pu(y − z)ϕ(z)χ s + dzdu s |x| dydsdx 0 Rd T ! Z Z ′ ′′ ′′′ = I1(T )+ I1 (T )+ I1 (T ), (3.81) where 1 ′ 1 I1(T ) = ... (3.82) log T Rd 1/α Z Z0 Z1≤|y|≤T 1 ′′ 1 I1 (T ) = ... (3.83) log T Rd 1/α Z Z0 Z|y|>T 1 ′′′ 1 I1 (T ) = ... (3.84) log T Rd Z Z0 Z|y|<1 Passing to polar coordinates in the integral with respect to y and making substitution (3.75) we obtain ′ I1(T ) 1 1/α r−1/α −1/α = p1(x − wT s ) Rd Z Z0 Z0 ZSd−1 1+β T (1−s) r u −γ/α −γ rd × pu(wT − z)ϕ(z)χ + s dzdu s |x| T σ(dw)drdsdx. 0 Rd T ! Z Z We substitute z′ = T −rz,u′ = uT −rα, use (3.20) and (2.14), arriving at 1 1/α ′ −1/α r−1/α 1+β −γ/α −γ I1(T )= p1(x − ws T )(hT (r,s,w)) s |x| σ(dw)drdsdx, Rd Z Z0 Z0 ZSd−1 (3.85) where T 1−rα(1−s) rd r rα−1 hT (r,s,w)= pu(w − z)T ϕ(zT )χ(s + uT )dzdu. (3.86) Rd Z0 Z It is clear that on the set of integration one should have ∞ lim hT (r,s,w) = ϕ(z)dz pu(w)duχ(s) T →∞ Rd Z Z0 = Cd,α ϕ(z)dzχ(s), (3.87) Rd Z where Cd,α is given by (1.13), which should yield 1 1+β ′ 1+β 1 −γ −γ/α 1+β lim I1(T )= Cd,α σ(Sd−1) p1(x)|x| dx s χ (s)ds ϕ(z)dz . (3.88) T →∞ α Rd 0 Rd Z Z Z 26 However, (3.87) and (3.88) need a justification. It is easy to see that the first integral in (3.86) ∞ can be replaced by 0 du. Since R rd r lim pu(w − z)T ϕ(zT )dz = pu(w) ϕ(z)dz, T →∞ Rd Rd Z Z it is clear that in order to prove (3.87) it suffices to show that rd r sup sup pu(w − z)T ϕ(zT )dz Rd T>2 w∈Sd−1 Z is integrable in u. This is clear for u ≥ 1 since d>α, and for u < 1 we argue similarly as in (3.39) obtaining an integrable bound C1pu(w/2) + C2. In the same way one shows that hT (r,s,w) ≤ C. This together with (3.87) easily implies (3.88). ′′′ Next, it is easy to see that for I1 defined by (3.84) we have C I′′′(T ) ≤ → 0. (3.89) 1 log T A little more work is needed to prove that also ′′ lim I1 (T )=0. (3.90) T →∞ By (3.83), T 1+β ′′ C I1 (T ) ≤ pu(y − z)ϕ(z)dzdu dy log T 1/α Rd Z|y|>T Z0 Z ≤ C1(R1(T )+ R2(T )), where 1+β 1 T R1(T ) = pu(y − z)ϕ(z)dzdu dy, 1/α log T |y|>T 1/α 0 |z|≤ T Z Z Z 2 ! 1+β 1 T R2(T ) = pu(y − z)ϕ(z)dzdu dy. 1/α log T |y|>T 1/α 0 |z|> T Z Z Z 2 ! We have 1+β 1 T y 1+β R1(T ) ≤ pu du dy ϕ(z)dz log T |y|>T 1/α 0 2 Rd Z Z Z C 1 y 1+β = pu du dy, log T |y|>1 0 2 Z Z after obvious substitutions and using (2.14). Hence lim R1(T ) = 0 by (3.23). Furthermore, T →∞ 1+β C T |z|2ϕ(z) R2(T ) ≤ pu(y − z) dzdu dy 1/α 2/α log T |y|>T 1/α 0 |z|> T T Z Z Z 2 ! C ≤ 1 , T (2/α)(1+β) 27 since, under (2.14), 1+β 1 T sup pu(y − z)ϕ1(z)dzdu dy < ∞, log T Rd Rd T Z Z0 Z d 2 for any ϕ1 ∈ S(R ) by (3.33) in [BGT4] (in our case ϕ1(z)= |z| ϕ(z)). This proves (3.90), and by (3.81)-(3.84), (3.88) and (3.89) we have established (3.12). To prove (3.13) we use (3.14), which, after standard substitutions, gives I2(T ) −1/(1+β) −2/(1+β)+2γ/α(1+β)+2−d/α−γ/α −2/(1+β) ≤ CHT T (log T ) fγ(y)ϕT (y)(f ∗ ϕT )(y)dy Rd Z (see (2.15), (3.22), (3.29), (3.34)). By (3.31) we have e e −2/(1+β)+2γ/α(1+β)+2−d/α−γ/α I2(T ) ≤ C1T sup f ∗ ϕT (y). (3.91) y The assumptions (2.14) and γ<α imply that e 2 2γ d γ 1 − − 2+ + = θ 1+ β α(1 + β) α α β for some θ> 1. Observe that f ∈ Lq(Rd), 1 ≤ q < 1+ β, (3.92) by (3.23), (3.24) and (2.14). Fix q > 1 such that (1 + β)/θ −θ/β+((1+β)/β)(1/q) I2(T ) ≤ C1T ||ϕ||p → 0 as T →∞, by assumption on q. To prove (3.16) we use (3.18) which, by an analogous argument as for I2, gives −1+γ/α−(1+β)/β 1+β J1(T ) ≤ CT ||f ∗ (ϕT (f ∗ ϕT ))||1+β. (3.93) Taking q < 1+ β sufficiently close to 1 + β and using (3.92), the Young and H¨older inequalities e e can be applied to conclude that, by (3.45), 1+β r ||f ∗ (ϕT (f ∗ ϕT ))||1+β ≤ O(T ), where r< 1 − γ/α +(1+ β)/β; we omit details. This and (3.93) yield (3.16). e e To prove (3.17) we write an estimate similar to (3.50), namely, β(γ/α−1) J2(T ) ≤ CT R(T ), (3.94) where R(T ) is defined by (3.51). In the present case (3.52) does not hold, but similarly as before, using the Young inequality, (3.92) and (3.45) it can be shown that R(T ) = O(T ε) for 28 any ε > 0. This and (3.43) imply (3.17) since γ<α. The proof of part (a) of the theorem is complete. Sketch of the proof of case (b) (i) We repeat the argument as in (3.81) -(3.86) with FT given by (2.16). Again, it can be ′ ′ shown that the only significant term is I1(T ), i.e., limT →∞ I1(T ) = limT →∞ I1(T ). In order to derive this limit, in (3.85) we substitute s′ = sT 1−rα, obtaining 1/α T 1−rα ′ 1 1/α −1+rα 1+β −1 −α I1(T )= p1(x − ws )(hT (r, sT ,w)) s |x| σ(dw)dsdrdx. log T Rd Z Z0 Z0 ZSd−1 T 1−rα It is easy to see that the limit remains the same if the integral 0 ...ds is replaced by 1−rα T ′ ′ ′ 1 ...ds. Let I1(T ) denote I1(T ) after this change. Next, weR substitute s = log s/ log T and we have R 1/α e 1−rα ′ −s/α −α s−1+rα 1+β I1(T )= p1(x − wT )|x| (hT (r, T ,w)) dxdsσ(dw)dr. Rd Z0 ZSd−1 Z0 Z By (3.86),e it is clear that on the set of integration s−1+rα lim hT (r, T ,w)= Cd,α ϕ(z)drχ(0). T →∞ Rd Z This shows that we should have ′ lim I1(T ) = lim I1(T ) T →∞ T →∞ 1/α 1+β 1+β e −d 1+β = Cd,α σ(Sd−1) p1(x)|x| dx (1 − rα)dr ϕ(z)dz χ (0), (3.95) Rd 0 Rd Z Z Z and this passage to the limit can be indeed justified. The right-hand side of (3.95) is equal to log Eexp{−ChX,ϕ ⊗ ψi}, where X(= Kλϑ) is the limit process defined in the theorem. We skip the remaining parts of the proof. (ii) In (3.9)e we substitute u′ = s − u and then s′ = T − s, obtaining ′ ′′ I1(T )= I1(T )+ I1 (T ), where T ′ 1 −1/α −d/α −γ I1(T ) = p1((x − y)s )s |x| log T 1 Rd Rd Z Z Z 1+β T −s u + s × pu(y − z)ϕ(z)χ dzdu dydxds 0 Rd T Z Z and 1 ′′ 1 I1 (T )= . . . dydxds. log T Rd Rd Z0 Z Z 29 It can be shown that ′′ lim I1 (T )=0. T →∞ ′ ′ −1/α ′ −1/α ′ In I1(T ) we substitute x = xs ,y = ys ,u = u/s, and use (3.20), which gives T ′ 1 −γ −(d/α)β−γ/α+(1+β) I1(T ) = p1(x − y)|x| s log T Rd Rd Z1 Z Z T/s−1 1+β −1/α s(u + 1) × pu(y − zs )ϕ(z)χ dzdu dydxds. 0 Rd T Z Z By (2.18), s−(d/α)β−γ/α+1+β = s−1, so, the substitution s′ = log s/ log T yields 1 ′ −γ I1(T ) = p1(x − y)|x| Rd Rd Z0 Z Z − 1+β T 1 s−1 −s/α s−1 × pu(y − zT )ϕ(z)χ ((u + 1)T dzdu dydxds. Rd Z0 Z ! (3.96) It is now seen that one should have ′ lim I1(T ) = lim I1(T ) T →∞ T →∞ 1+β 1+β −γ −(d−α)(1+β) 1+β = Cα,d p1(x − y)|x| |y| dxdy ϕ(z)dz χ (0) (3.97) Rd Rd Rd Z Z Z Note that the integrals are finite by (3.21), (3.28) and (2.18). The justification of (3.97) requires some work, but we omit it for brevity. (iii) As d = γ, we must keep the measure µγ in its original form (1.8). Arguing as in the proof of (ii) and taking into account (2.16), instead of (3.96) we obtain 1 sd/α ′ 1 T I1(T )= p1(x − y) s/α d log T Rd Rd 1+ |xT | Z0 Z Z − 1+β T 1 s−1 −s/α s−1 × pu(y − zT )ϕ(z)χ((u + 1)T )dzdu dydxds. Rd Z0 Z ! Since 1 T sd/α 1 lim p1(x − y) s/α d dx = s σ(Sd−1)p1(y), T →∞ log T Rd 1+ |xT | α Z it can be shown, with some effort, that ′ lim I1(T ) = lim I1(T ) T →∞ T →∞ 1+β 1+β 1 −(d−α)(1+β) 1+β = Cα,d σ(sd−1) p1(y)|y| dy ϕ(z)dz χ (0). 2α Rd Rd Z Z 30 Again, we omit the remaining parts of the proof. 3.7 Proof of Theorem 2.6 We only give an outline of the proof. The following lemma is constantly used. Lemma Let ϕ ∈ S(Rd),ϕ ≥ 0. (a) If d>α(2 + β)/(1 + β), then the functions Gϕ,G(Gϕ)1+β and G(Gϕ)2 are bounded. (b) If d > α(1 + β)/β, then additionally (Gϕ)1+β and (G(Gϕ)1+β)1+β are integrable (and bounded). (c) If α<γ ≤ d and d>α(2 + β)/β − γ/β, then additionally to the properties in (a), 1+β 1 G(Gϕ) (x) γ dx < ∞. Rd 1+ |x| Z This Lemma follows easily from (1.12) and (3.25)-(3.28). −γ Proof of part (a) of the theorem. As before we consider µγ(dx) = |x| dx. In (3.9) we substitute u′ = s − u, then s′ =(T − s)/T and, finally, x′ = xT 1/αs−1/α, obtaining I1(T ) 1+β 1 T (1−s) −1/α −1/α u −γ/α −γ = p1(x − ys T ) Tuϕ(y)χ s + du s |x| dydsdx Rd 0 Rd 0 T ! Z Z Z Z (see (2.24)). It is easily seen that by part (b) of the Lemma we have 1 −γ −γ/α 1+β 1+β lim I1(T )= p1(x)|x| dx s χ (s)ds (Gϕ(y)) dy. (3.98) T →∞ Rd Rd Z Z0 Z For β < 1 this is exactly log Eexp{−ChX,ϕ ⊗ ψi}, where X is the limit process described in the theorem. Moreover, in this case (3.14) and boundedness of Gϕ easily imply e (1−γ/α)(1−2/(1+β)) I2(T ) ≤ C T → 0. For β = 1 we use (3.10) and (3.4), obtaining V I (T )= I′ (T ) − I′′(T ) − I′′′(T ), 2 2 2 2 2 where T T ′ HT −γ u I2(T ) = 2 ps(x − y)|x| dxϕ(y)χ(s) Tu−sϕ(y)χ dudyds, FT 0 Rd Rd s T Z Z T Z Z ′′ HT I2 (T ) = 2 pT −s(x − y)ϕ(y)χT (T − s) FT Rd 0 Rd Zs Z Z −γ × Ts−u(ϕχ(T − u)vT (·,u))(y)|x| dudydsdx 0 Z T s ′′′ HT 2 −γ I2 (T ) = 2 pT −s(x − y)ϕ(y)χT (T − s) Ts−u(vT (·,u))(y)|x| dudydsdx. F Rd Rd T Z Z0 Z Z0 31 Substituting u′ = u − s,s′ = s/T , and then x′ = xT 1/αs1/α and using part (a) of the Lemma and (2.24) we have 1 ′ −γ −γ/α 2 lim I2(T )= p1(x)|x| dx s χ (s)ds ϕ(y)Gϕ(y)dy. (3.99) T →∞ Rd Rd Z Z0 Z Applying (3.5), (2.24) and the Lemma above we get T ′′ CHT −γ I2 (T ) ≤ 3 ps(x − y)|x| ϕ(y)G(ϕGϕ)(y)dydsdx F Rd Rd T Z Z0 Z −(1/2)(1−γ/α) ≤ C1T → 0, and, analogously, T ′′′ CHT −γ 2 I2 (T ) ≤ 3 ps(x − y)|x| ϕ(y)G((Gϕ) )(y)dydsdx F Rd Rd T Z Z0 Z −(1/2)(1−γ/α) ≤ C2T → 0. This and (3.98), (3.99) imply that for β = 1 the limit of (V/2)I1(T )+ I2(T ) is exactly log Eexp{−ChX,ϕ ⊗ ψ)}. Similar estimations, together with the Lemma, yield (3.16) and (3.17). This completes the proof of part (a) of the theorem. e Proof of part (b) of the theorem. Following the general scheme one can show −α 1+β 1+β lim I1(T ) = p1(x)|x| dxχ (0) (Gϕ) (y)dy, T →∞ Rd Rd Z Z −α 2 lim I2(T ) = cβ p1(x)|x| dxχ (0) ϕ(y)Gϕ(y)dy, T →∞ Rd Rd Z Z and (3.16) and (3.17) (recall that cβ is defined by (2.26)). This is accomplished by an argument T similar to the one used in part (a). Due to the criticality (γ = α), the integrals 0 ...ds in (3.9)-(3.11) require a different treatment. They are split into 1 ...ds + T ...ds; the first 0 1 R summand converges to zero, and in the second one we use the substitution s′ = log s/ log T . R R Here, again, we use repeatedly the Lemma above together with the easily checked fact that T 1 −α sup Tsh(x)|x| dsdx < ∞ log T Rd T>2 Z Z0 for any integrable and bounded function h (recall that d>α). We omit details. Proof of part (c) of the theorem. Recall that the case γ>d has been proved in [BGT6]. For α<γ ≤ d we use the Lemma (part (c) is particularly important). We show 1+β 1+β lim I1(T ) = G(Gϕ) (x)µγ(dx)χ (0), T →∞ Rd Z 2 lim I2(T ) = cβ G(ϕGϕ)(x)µγ(dx)χ (0), T →∞ Rd Z 32 −γ (3.16) and (3.17). Here µγ is either given by (1.8) or, for γ 3.8 Proof of Theorem 2.8 The proof is based on two general lemmas which, hopefully, are of interest by themselves. Lemma A. Let Y be an (d,α,β)-superprocess with Y0 = µ and N be the empirical process of the corresponding branching particle system. If for any bounded Borel set A ⊂ Rd, ∞ P Yt(A)dt < ∞ =1, (3.100) Z0 then also ∞ P Nt(A)dt < ∞ =1. (3.101) Z0 Proof of Lemma A. Let ζ denote the standard α-stable L´evy process in Rd, and let ξ be a Markov process with semigroup t Stϕ(x)= Ex exp − ψ(ζs)ds ϕ(ζt) , Z0 ∞ Rd where ψ is a fixed element of CK ( ) (bounded support), ψ ≥ 0. The process ξ takes values Rd t in ∪ {†}, where † is a cemetery point where it remains after killing by exp{− 0 ψ(ζs)ds}. The infinitesimal generator of ξ is R Aϕ(x)=(∆α − ψ(x))ϕ(x). ψ Rd ψ Let Y be a superprocess in constructed from ξ and (1 + β)-branching, with Y0 = µ. The Laplace functional of its occupation time is given by t ψ ψ ∞ Rd Eexp − hYs ,ϕids = exp{−hµ,uϕ(t)i}, ϕ ∈ CK ( ), ϕ ≥ 0, (3.102) Z0 ψ where uϕ(x, t) is the unique (mild) solution of ∂ V uψ(x, t) = (∆ − ψ(x))uψ(x, t) − (uψ(x, t))1+β + ϕ(x), (3.103) ∂t ϕ α ϕ 1+ β ϕ ψ uϕ(x, 0) = 0 (cf. (1.6)). The Laplace functional of the occupation time of the process N is given by t ∞ Rd Eexp − hNs,ϕids = exp{−hµ, vϕ(t)i}, ϕ ∈ CK ( ), ϕ ≥ 0, (3.104) Z0 33 where vϕ(x, t) is the unique solution of ∂ V v (x, t) = ∆ v (x, t) − (v (x, t))1+β + ϕ(x)(1 − v (x, t)) ∂t ϕ α ϕ 1+ β ϕ ϕ V = (∆ − ϕ(x))v (x, t) − (v (x, t))1+β + ϕ(x), (3.105) α ϕ 1+ β ϕ vϕ(x, 0) = 0 (cf. (1.4)). Equations (3.103) and (3.105) coincide for ϕ = ψ, hence, from (3.102), (3.104), t t ψ ψ Eexp − hYs , ψids = Eexp − hNs, ψids = exp{−hµ,uψ(t)i}. (3.106) Z0 Z0 The superprocesses Y ≡ Y 0 and Y ψ are obtained as (high-density/short-life/small-particle) limits of the same process N, removing first the killed particles in the case of Y ψ. Hence t t ψ 0 Eexp − hYs ,ϕids ≥ Eexp − hYs ,ϕids for all t, (3.107) Z0 Z0 for any ϕ ≥ 0, in particular for ϕ = ψ, hence, from (3.106), (3.107), and (3.102) with ψ = 0, ψ 0 exp{−hµ,uψ(t)i} ≥ exp{−hµ,uψ(t)i} for all t. (3.108) Taking µ = δx in (3.102) we see that ψ ψ 0 0 uψ(x, t) ր uψ(x) and uψ(x, t) ր uψ(x) as t ր∞, hence, from (3.108), ψ 0 hµ,uψi≤hµ,uψi. (3.109) From (3.100), (3.106), (3.109), ∞ 0 1 = P hYt, ψidt < ∞ = lim exp{−hµ,uθψi} 0 θց0 Z ∞ θψ ≤ lim exp{−hµ,uθψi} = P hNt, ψidt < ∞ , θց0 Z0 so (3.101) is satisfied for any bounded set A ⊂ Rd and the lemma is proved. ✷ Lemma B. Let N be the empirical process of the (d,α,β)-branching particle system with locally finite initial intensity measure µ. If (3.101) is satisfied for any bounded set A ⊂ Rd, then P [sup{t : Nt(A) > 0} < ∞] = 1 (3.110) for any bounded set A. d Proof of Lemma B. Let BR be a closed ball in R with radius R centered at the origin. Let (ti, xi, τi), i = 1, 2,..., be a sequence of random vectors defined as follows for any realization 34 of the branching particle system. First we exclude all the particles which start inside BR at time 0 and their progenies. Let t1 be the first time any of the remaining particles enters BR, x1 is the entry point, and τ1 is the occupation time of the closed ball B1(x1) of radius 1 centered at x1 by the tree generated by the entered particle. We exclude this tree from further consideration. Let t2 be the first time after t1 that any of the remaining particles enters BR, with x2 and τ2 defined analogously as above; and so on. Let η denote the total number of first entries (ti, xi, τi), i = 1,...,η. We will show that η < ∞ a.s.. Suppose to the contrary that η ∞ η P [η = ∞] > 0. By construction, i=1 τi ≤ 0 Nt(BR+1)dt, hence i=1 τi < ∞ a.s. by (3.101). By the strong Markov property and homogeneity of the motion, conditioned on {η = ∞} the P R P random variables τi are i.i.d.. Hence η P τi = ∞|η = ∞ =1, i=1 h i X ∞ and this is a contradiction since, as observed above, P [ i=1 τi = ∞] = 0. Going back to the particles that start inside BR, there are only finitely many of them since P µ(BR) < ∞. In conclusion, with probability 1 only finitely many initial particles generate trees that contribute to the occupation time of any given bounded set, and all those trees become extinct a.s. in finite time by criticality of the branching. So (3.110) is proved. ✷ Now, to prove Theorem 2.8 it suffices to observe that under its assumptions the corre- sponding superprocess Y suffers local extinction by Theorem 3β of [I2], hence (3.100) is clearly satisfied and the theorem follows immediately from the lemmas. 3.9 Proof of Proposition 2.9 First observe that it suffices to prove convergence of finite-dimensional distributions. Indeed, in the proof of Theorem 2.1(a) we have shown tightness of XT = ZT − EZT , and the presence of high density was not relevant in that proof. On the other hand, from Proposition 2.1 of [BGT5] it follows easily that the family of deterministic processes (EhZT ,ϕi)T ≥1 is tight in C([0, τ], R),τ > 0. Hence tightness of ZT follows. Without loss of generality we assume that τ = 1. Fix 0 ≤ t1 < t2 < ... < tn ≤ d 1,ϕ1,...,ϕn ∈ S(R ), and we may additionally assume that ϕ1,...,ϕn ≥ 0. In order to show ⇒f convergence we prove that n −γ lim Eexp − hZT (tk),ϕki = exp − v(x, 1)|x| dx , (3.111) T →∞ ( ) Rd Xk=1 Z where v satisfies (2.36) with ψ given by (2.37) for θk = Rd ϕk(y)dy (as explained in [Ta], the solution of (2.36) is unique.) −γ R For simplicity we consider µγ(dx) = |x| dx (it will be clear that the limit is the same as for µγ given by (1.8)). Also, to simplify the notation we take ϕ1 = ... = ϕn = ϕ. Essentially the same argument can be carried out in the general case. As in [Ta] and [BGT4] (the possibility to pass from space-time random variable to the 35 present situation) we have n −γ Eexp − hZT (tk),ϕi = exp − vT (x, T )|x| dx , (3.112) Rd ( k=1 ) Z X n where v satisfies (3.4) with χ(t) = k=1 11[0,tk](t), χT (t) = χ(t/T ). Formula (3.112) is an analogue of (3.7), and its form is simpler since now we do not subtract the mean. The right-hand side of (3.112) canP be written as −γ exp − hT (x, 1)|x| dx , Rd Z where d/α−γ/α 1/α hT (x, t)= T vT (xT , T t), 0 ≤ t ≤ 1. (3.113) 1 d −γ To prove (3.111) it suffices show that hT (·, 1) converges to v(·, 1) in L (R , |x| dx); in fact, we will prove that 1 d −γ hT → v in C([0, 1], L (R , |x| dx)). (3.114) By (3.113), (3.4), (2.33) and (3.20) we have t d/α 1/α hT (x, t) = Tt−sT ϕ(T ·)(x)χ(1 − s)ds 0 Z t γ/α 1/α − T Tt−s(ϕ(T ·)hT (·,s))(x)χ(1 − s)ds Z0 V t − T (h (·,s))1+β(x)ds. (3.115) 1+ β t−s T Z0 In particular, this implies that t d/α 1/α hT (x, t) ≤ CT Tsϕ(T ·)ds. (3.116) Z0 Let RT (x) t t d/α 1/α = sup Tt−s(T ϕ(T ·))(x)χ(1 − s)ds − pt−s(x)χ(1 − s)ds ϕ(y)dy . (3.117) Rd t≤1 Z0 Z0 Z (compare with the first formula on page 851 of [Ta]). We will show that −γ lim RT (x)|x| dx =0. (3.118) T →∞ Rd Z Applying the usual substitutions and the fact that pu(x) is a decreasing function of |x| we obtain 1 −γ −1/α −γ RT (x)|x| dx ≤ C ϕ(y) (pu(x − T ) − pu(x))du |x| dydx Rd Rd Rd Z Z Z Z0 ≤ C ϕ(y) |f(x − T −1/αy) − f(x)|dxdy Rd Rd Z Z 1/p 1/q + C ϕ(y)|x|−γpdydx ϕ(y)|f(x − T −1/αy) − f(x)|qdydx , Rd Rd Z|x|≤1 Z Z|x|≤1 Z 36 where f is defined by (3.22), and p,q > 1 are such that γp < d, (d − α)q < d, 1/p +1/q = 1 (such p and q exist since γ<α t γ/α 1/α −γ lim sup T Tt−s(ϕ(T ·)χ(1 − s)hT (·,s)(x) |x| dx =0. (3.119) T →∞ Rd Z t≤1 Z0 (3.116) and (3.31) imply that the expression under the lim can be estimated by γ/α 1/α γ/α 1/α T fγ(y)ϕ(T y)(f ∗ ϕT )(y)dy ≤ T ||ϕ(T ·)|| 1+β ||f||1+β||ϕ||1 Rd β Z ≤ CT γ/α−(d/α)β/(1+β) → 0 e (we have used (2.33) and f ∈ L1+β by (3.23) and (3.24)). To prove (3.114) we check the Cauchy condition, i.e., −γ J(T1, T2) := sup |hT1 (x, t) − hT2 (x, t)||x| dx → 0 as T1, T2 →∞. (3.120) Rd Z t≤1 Using (3.115), (3.118) and (3.119) we have V J(T , T ) ≤ J (T , T )+ J (T , T ), (3.121) 1 2 1 1 2 1+ β 2 1 2 where lim J1(T1, T2) = 0 and T1,T2→∞ t 1+β 1+β −γ J2(T1, T2)= sup Tt−s|hT1 (·,s) − hT2 (·,s)|(x)ds|x| dx. Rd Z t≤1 Z0 By (3.31) and the H¨older inequality, 1+β 1+β J2(T1, T2) ≤ C sup |hT1 (y, t) − hT2 (y, t)|dy Rd Z t≤1 β β ≤ C1 sup |hT1 (·, t) − hT2 (·, t)| sup |hT1 (·, t)| + sup |hT2 (·, t)| t≤1 1+β t≤1 1+β t≤1 1+β! Using (3.116) and the fact that f ∈ L1+ β it is easily seen that sup sup hT (·, t) < ∞. T ≥1 t≤1 1+β To show (3.120) it suffices to prove that lim sup |hT1 (·, t) − hT2 (·, t)| =0. (3.122) T ,T →∞ 1 2 t≤1 1+β 37 This can be derived in a similar way as in [Ta] (see (2.21) and subsequent estimates therein). The only difference is that the term corresponding to I2(T ) in [Ta] requires a slightly more delicate treatment; in our case it has the form s 1+β (γ/α)(1+β) 1/α T sup Ts−u(ϕ(T ·)hT1 (·,u))(x)du dx. Rd Z s≤1 Z0 Using (3.116) and the H¨older inequality this is estimated by (d/α)(1+β) 1/α 1+β (γ/α)(1+β) 1+β 1/α 1+β 1+β 1+β T ||f ∗ (ϕ(T ·)(f ∗ ϕT ))||1+β ≤ T ||f||1+β||ϕ(T ·)||p ||f||q ||ϕ||1 , where q = d/(d − α + ε),p = d/(α − ε), and ε> 0 is such that γ<α − ε. Then the right-hand e side is not bigger than CT ((1+β)/α)(γ+ε−α), which tends to zero as T →∞. Combining (3.120), (3.118), (3.119) and (3.122), it is seen that one can pass to the limit in (3.115) letting T →∞, thus obtaining that the limit of hT satisfies (2.36). This proves (3.114) and completes the proof of the Proposition. ✷ Proof of Theorem 2.11 The proof is similar to those for the particle system, starting from an equation analogous to (3.7)-(3.8), where XT is now defined for the occupation time fluctuation process (1.10) corresponding to the (d,α,β,γ) superprocess Y , and equation (3.4) is replaced by e t V v (x, t)= T ϕ χ (T − u) − v1+β(·,u) (x)du, (3.123) T t−u T T 1+ β T Z0 where the term I2(T ) in equation (3.8), given by (3.10), does not appear. This reflects the fact that comparing the log-Laplace equations (1.4) for the particle system and (1.6) for the superprocess, the term −ϕvϕ is missing in (1.6). (Equation (3.123) can be obtained from (3.4) by the same limiting procedure that yields the superprocess from the branching particle system. An equation analogous to (3.7) for the superprocess can be derived from continuous dependence of the occupation time process with respect to the superprocess, and continuity of the mapping C([0, τ], S′(Rd) ∋ x 7→ x ∈ S′(Rd+1) in (1.11) [BGR].) It follows that the results for the superprocess are the same as those for the particle system, except in the cases where I2(T ) has a non-zero limit, and to obtaine the results in those cases it suffices to delete those non-zero limits. Therefore the limits in Theorems 2.1 and 2.5 are the same for the superprocess, and for those in Theorem 2.6, cβ = 0 in all cases. ✷ Acknowledgment. We thank Professors Zenghu Li and Xiaowen Zhou for useful discussions on the superprocess. We are grateful for the hospitality of the Institute of Mathematics, Na- tional University of Mexico, where this work was partially done. References [B] P. Billingsley, Convergence of Probability Measures, Wiley, New York, 1968. 38 [BZ] M. Birkner and L. Z¨ahle, A functional CLT for the occupation time of state-dependent branching random walk, Ann. Probab. 35(6), (2007), 2063-2090. [BGR] T. Bojdecki, L.G. Gorostiza and S. Ramaswami, Convergence of S′-valued processes and space-time random fields, J. Funct. Anal. 66, (1986), 21-41. [BGT] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, Sub-fractional Brownian motion and its relation to occupation times, Stat. Prob. Lett. 69, (2004), 405-419. [BGT1] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, Limit theorems for occupation time fluctuations of branching system I: Long-range dependence, Stoch. Proc. Appl. 116, (2006), 1-18. [BGT2] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, Limit theorems for occupation time fluctuations of branching systems II: Critical and large dimensions, Stoch. Proc. Appl. 116, (2006), 19-35. [BGT3] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, A long-range dependence stable process and an infinite-variance branching system, Ann. Probab. 35 (2), (2007), 500-527. [BGT4] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, Occupation time fluctuations of an infinite-variance branching system in large dimensions, Bernoulli 13 (1), (2007), 20-39. [BGT5] T. Bojdecki, L.G. Gorostiza and Talarczyk, Occupation time limits of inhomogeneous Poisson systems of independent particles, Stoch. Proc. Appl. 118, (2008), 28-52. [BGT6] T. Bojdecki, L.G. Gorostiza and A. Talarczyk, Self-similar stable processes arising from high-density limits of occupation times of particle systems, Potential Analysis 28 (1), (2008), 71-103. [Br] L. Breiman, Probability, Addison-Wesley, Reading, 1968. [CS] S. Cohen and G. Samorodnitsky, Random rewards, fractional Brownian local times and stable self-similar processes, Ann. Applied Probab. 16 (3), (2006), 1432-1461. [CG] J.T. Cox and D. Griffeath, Occupation times for critical branching Brownian motions, Ann. Probab. 13, (1985), 1108-1132. [D] D.A. Dawson, Measure-valued Markov processes, Lect. Notes Math. 1541, (1993), 1-260. Springer. [DGW] D.A. Dawson, L.G. Gorostiza and A. Wakolbinger, Occupation time fluctuations in branching systems, J. Theoret. Probab. 14, (2001), 729-796. [DP1] D.A. Dawson and E.A. Perkins, Historical Processes, Mem. Amer. Math. Soc. 93, (1991), No. 454. [DP2] D.A. Dawson and E.A. Perkins, Measure-valued processes and renormalization of branch- ing particle systems, Stochastic Partial Differential Equations: Six Perspectives, Math. Surveys and Monographs 64, (1999), 45-106, Amer. Math. Soc. 39 [DR] J.-D. Deuschel and J. Rosen, Occupation time large deviations for critical branching Brownian motion, super-Brownian motion and related processes, Ann. Probab. 26, (1998), 602-643. [DF] Z. Dong and S. Feng, Occupation time processes of super-Brownian motion with cut-off branching, J. Appl. Prob. 41, (2004), 984-997. [DOT] P. Doukhan, G. Oppenheim and M.S. Taqqu (editors), Theory and Applications of Long-Range Dependence, Birkh¨auser, 2003. [EK] J. Englander and A.E. Kyprianou, Local extinction versus local exponential growth for spatial branching processes. Ann. Probab. 32, (2004), 78-99. [E] A.M. Etheridge, An Introduction to Superprocesses, Univ. Lect. Series 20 (2000), Amer. Math. Soc. [FG] K. Fleischmann and J. G¨artner, Occupation time processes at a critical point, Math. Nachr. 125, (1986), 275-290. [FVW] K. Fleischmann, V. Vatutin and A. Wakolbinger, Branching system with long living particles at the critical dimension, Theory Probab. Appl. 47, (2002), 417-451. [GLM] L.G. Gorostiza and J.A. L´opez-Mimbela, An occupation time approach for convergence of measure-valued processes, and the death process of a branching system, Statist. Probab. Lett. 21, (1994), 59-67. [GNR] L. G. Gorostiza, R. Navarro and E. Rodrigues, Some long-range dependence processes arising from fluctuations of particle systems, Acta Appl. Math. 86, (2005), 285-308. [GW] L.G. Gorostiza and A. Wakolbinger, Persistence criteria for a class of critical branching particle systems in continuous time, Ann. Probab. 19, (1991), 266-288. [H1] C.C. Heyde, On defining long-range dependence, J. Appl. Probab. 34, (1997), 939-944. [H2] C.C. Heyde, On modes of long-range dependence, J. Appl. Probab. 39, (2002), 882-888. [HJ] B. Hambly and L. Jones, Number variance from a probabilistic perspective: Infinite sys- tems of independent Brownian motions and symmetric α-stable processes, Electron. J. Probab. 12, (2007), 862-887. [H] W.M. Hong, Longtime behavior of the occupation time of super-Brownian motion with random immigration, Stoch. Proc. Appl. 102, (2002), 43-62. [HV] C. Houdr´eand J. Villa, An example of infinite-dimensional quasi-helix, in Stochastic Models, Contemp. Math. 366 (2003), Amer. Math. Soc., 195-201. [I1] I. Iscoe, A weighted occupation time for a class of measure-valued branching processes, Probab. Theor. Relat. Fields 71, (1986), 85-116. [I2] I. Iscoe, On the supports of measure-valued critical branching Brownian motion, Ann. Probab. 16 (1), (1988), 200-221. 40 [IL] I. Iscoe and T.Y. Lee, Large deviations for occupation times of measure-valued Brownian motions, Stoch. Stoch. Reports 45, (1993), 177-209. [KT] I. Kaj and M.S. Taqqu, Convergence to fractional Brownian motion and to the Telecom process: the integral representation approach, in Brazilian School of Probability, 2007. (To appear). [K] A. Klenke, Clustering and invariant measures for spatial branching models with infinite variance, Ann. Probab. 26, (1998), 1057-1087. [LR] T.Y. Lee and B. Remillard, Large deviations for three dimensional super-Brownian mo- tion, Ann. Probab. 23, (1995), 1755-1771. [LT] J.B. Levy and M.S. Taqqu, Renewal reward processes with heavy-tailed inter-renewal times and heavy tailed rewards, Bernoulli 6 (1), (2000), 23-44. [LZ] Z. Li and X. Zhou, Distribution and propagation properties of Dawson-Watanabe super- processes, (preprint). [MY] M. Maejima and K. Yamamoto, Long-memory stable Ornstein-Uhlenbeck processes, Elec- tron, J. Probab. 8, (2003), 1-18. [MR] S. M´el´eard and S. Roelly, An ergodic result for critical spatial branching processes. In Stochastic Analysis and Related Topics (Silivri, 1990). Progr. Probab. 31, (1992), 333-341, Birkh¨auser. [M1] P. Milo´s, Occupation time fluctuations of Poisson and equilibrium finite variance branch- ing systems, Probab. Math. Statist. 27 (2), (2007), 181-203. [M2] P. Milo´s, Occupation time fluctuations of Poisson and equilibrium branching systems in critical and large dimensions, Probab. Math. Statist., (to appear). [M3] P. Milo´s, Occupation time fluctuation limits of infinite variance equilibrium branching systems, (preprint), Math. ArXiv. PR 0802.0187. [Mi] I. Mitoma, Tightness of probabilities on C([0, 1]; S′) and D([0, 1]; S′), Ann. Probab. 11, (1983), 989-999. [P1] E.A. Perkins, Polar sets and multiple points for super-Brownian motion, Ann. Probab. 18, (1989), 453-491. [P2] E.A. Perkins, Dawson-Watanabe superprocesses and measure-valued diffusions, Lect. Notes Probab. Math. Statist. 1781, (2002), 125-324, Springer. [PTL] V. Pipiras, M.S. Taqqu and J.B. Levy, Slow, fast and arbitrary growth conditions for renewal reward processes when the renewals and the rewards are heavy-tailed, Bernoulli 10, (2004), 121-163. [RZ] J. Rosi´nski and T. Zak,˙ The equivalence of ergodicity and weak mixing for infinitely divisible processes, J. Theoret. Probab. 10, (1997), 73-86. 41 [S] G. Samorodnitsky, Long range dependence, Foundations and Trends in Stoch. Systems 1(3), (2006), 163-257. [ST] G. Samorodnitsky and M.S. Taqqu, Stable Non-Gaussian Processes, Chapman & Hall, New York, 1994. [Sh] Y. Shiozawa, Extinction of branching symmetric α-stable processes, J. Applied Probab. 43(4), (2006), 1077-1090. [Ta] A. Talarczyk, A functional ergodic theorem for the occupation time process of a branching system, Stat. Prob. Lett. 78 (7), (2008), 847-853. [T] M.S. Taqqu, Fractional Brownian motion and long-range dependence, in Theory and Ap- plications of Long-Range Dependence, Birkh¨auser, 2003, 5-38. [VW] V. Vatutin and A. Wakolbinger, Spatial branching populations with long individual lifetimes, Theory Probab. Appl. 43, (1999), 620-632. [Zh] M. Zhang, Functional central limit theorem for the super-Brownian motions with super- Brownian immigration, J. Theoret. Probab. 18, (2005), 665-685. [Z] X. Zhou, A zero-one law of almost sure local extinction for (1+β)-super-Brownian motion, Stoch. Proc. Appl., (to appear). 42