Arxiv:1909.04241V4

arXiv:1909.04241v4 [math.AG] 31 May 2021 of optimal ojcuefor conjecture S n[3 n ig hae.W anyflo h dao Tanaka of idea the twi follow the mainly that conjecture We We invariants. Vafa-Witten sheaves. twisted Higgs and [43] in ufcscmaigwt h euto aaaToa n[56, in Tanaka-Thomas of result the with comparing surfaces hsnwosraingvstepeito oml fVafa-W of formula SU prediction the gives observation new this n eiwi ie n[6.Ti hoyivle opigc coupling involves theory This [26]. in given is review a and uesmercYn-il hoyo el4-manifold real a on theory Yang-Mills supersymmetric s 1.1. smooth any for work should paper this in used techniques The ae st rvd ehdt en h SU the define to method a provide to is paper ebsoe mohpoetv ufcsb sn h ouis moduli the using by surfaces projective smooth over gerbes aaee fteuprhl plane half upper the of parameter ojcueo aaWte n[1.W td n en h twi the define and study We [61]. in Vafa-Witten of conjecture h -ult rdcsta h transformation the that predicts S-duality The G L M SU otepriinfnto ihLnlnsda ag group gauge dual Langlands with function partition the to ntecs fK ufcs n ftenvldsoeisi tha is discoveries novel the of one surfaces, K3 of case the In h agad ulguegopo h ag ru SU group gauge the of group gauge dual Langlands The µ Ñ ( neligasot rjciesurface projective smooth a underlying 2 r ( -ult ojcueo Vafa-Witten. of conjecture S-duality gre eedo h iadnme ftesurface the of number Picard the on depend -gerbes ) r / S = ) Z udmna lse ntemdl pcsaddfietetwist the spaces. moduli define the SU on and function Behrend spaces the sheav moduli and localization general the like on behave classes sheaves fundamental Higgs and sheaves twisted wse hae n hi ouiaentrlyue ostudy to used naturally are moduli their corresponding and sheaves Twisted ufcsi rm ak.W locnetr o te surfac other for conjecture also We f ranks. Vafa-Witten prime of in conjecture surfaces S-duality the prove and gerbes, mohpoetv ufcsuigtemdl pcsof spaces moduli the using surfaces projective smooth A µ nasurface a on 2 r BSTRACT gre naK surface K3 a on -gerbes ( SU = r ) / ( SO Z P r ) r 2 / fSU of ( epoieadfiiino aaaToa’ aaWte inv Vafa-Witten Tanaka-Thomas’s of definition a provide We . nrn w;adK ufcsi l h rm ranks. prime the all in surfaces K3 and two; rank in 3 Z ) r µ n[1 oml 41).W rv h -ult ojcueo conjecture S-duality the prove We (4.18)]. Formula [61, in emk hs rnfrain oepeiefloig[61, following precise more transformations these make We . ( r S r grei h rurgopo h ufcs eomto n o and Deformation surfaces. the of group Brauer the in -gerbe ONIGTITDSEVSADS-DUALITY AND SHEAVES TWISTED COUNTING ) edfieteSU the define we , iers othe to rise give Γ 0 ( S 4 H iedfeettitdVf-itnivrat.Teediff These invariants. Vafa-Witten twisted different give = ) Let . ( r " ) L / .I 1. SU Z Γ τ UFN JIANG YUNFENG 0 r Vf-itnivrat sn h wse nainsfr´ for invariants twisted the using invariants -Vafa-Witten ( b a d c ebifl eiwteSdaiycnetr of conjecture S-duality the review briefly We S : ( NTRODUCTION 4 = r ( ) = ) over r τ Ă 2 ) θ / π ´ Ñ 1 SL Z + SU C P r ( Vf-itnivrat n hc h S-duality the check and invariants -Vafa-Witten 2, 4 SL τ 1 ( and g µ π r rtepoetv ln nrn w n o K3 for and two rank in plane projective the or es. Z 2 r astepriinfnto o ag group gauge for function partition the maps ) sapiain o h agad ulgroup dual Langlands the for applications As gretitdsevsadHgssheaves. Higgs and sheaves twisted -gerbe ( i tdVf-itnivrat o all for invariants Vafa-Witten sted / 2, M ) . Z S sadHgssevs edfievirtual define We sheaves. Higgs and es etesubgroup the be G Z te o h agad ulguegroup gauge dual Langlands the for itten ntern w ae o 3surface, K3 a for case, two rank the In . dVf-itnivrat sn virtual using invariants Vafa-Witten ed 5 o h SU the for §5] 6] oedtiscnb on n[61], in found be can details More [61]. r ( onstants Vf-itnivrat.W rv the prove We invariants. -Vafa-Witten ) akof tack r L = ) 4 : G h setal trivial essentially the t h eididxtermfrthe for theorem period-index the urface. sSU is aaWte osdra4-manifold a consider Vafa-Witten . SU tdVf-itnivrat for invariants Vafa-Witten sted | Toa n[5,[6 odfiethe define to [56] [55], in -Thomas c * ( . r rat o eaegre over gerbes étale for ariants µ ( ) θ r r h agad ulgroup dual Langlands The . , gretitdsal sheaves stable twisted -gerbe ) g / srcinter fthe of theory bstruction ( obnda follows as combined Z 2 ) r Vf-itninvariants. -Vafa-Witten h anga fthis of goal main The . aaWte o K3 for Vafa-Witten f 3.Think §3]. µ r etale gre and -gerbes rn type erent µ r N τ -gerbes sthe as = µ r 4 - 2 YUNFENG JIANG

Γ The group 0(4) acts on H by aτ + b τ . ÞÑ cτ + d The group SL(2, Z) is generated by transformations 0 1 1 1 S = ´ ; T = . 1 0 0 1 From [61], invariance under T is the assertion that physics is periodic in θ with period 2π, and S is 2 2 equivalent at θ = 0 to the transformation g ( g )´1 originally proposed by Montonen and Olive 4π ÞÑ 4π [50]. One can check that T(τ) = τ + 1, and S(τ) = 1 . ´ τ For a smooth projective surface S, let Z(SU(r); τ) = Z(SU(r); q) be the partition function which counts the invariants of instanton moduli spaces, where we let q = e2πiτ. Similarly let Z(SU(r)/Zr; τ) be the partition function which counts the invariants of SU(r)/Zr-instanton moduli spaces. As pointed out in [61, §3], when some vanishing theorem holds, the invariants count the Euler characteristic of instanton moduli spaces. We will see a mathematical meaning of this vanishing. Now the S-duality predicts the following: Conjecture 1.1. The transformation T acts on Z(SU(r); q), and the S-transformation sends ω 1 ´ χ τ 2 (1.1.1) Z SU(r); = r 2 Z(SU(r)/Z ; τ). ´ τ ˘ i r for some ω, where χ := χ(S) is the topological Euler number of S. Usually ω = χ. This is Formula (3.18) in [61]. In mathematics we think Z(SU(r); τ) = Z(SU(r); q) as the partition function which counts the invariants of moduli space of vector bundles or Higgs bundles on S. Let 1 k η(q) = q 24 (1 q ) ´ kźě1 be the Dedekind eta function. Let Z(SU(r); q) = η(q)´w Z(SU(r); q), ¨ and we will see that Z(SU(r); τ) is the partition function of the moduli space of Gieseker stable Higgs p sheaves. Then S-duality predicts: p Conjecture 1.2.

1 ´ χ (1.1.2) Z SU(r); = r 2 Z(SU(r)/Z ; τ) ´ τ ˘ r p p 1 0 This is Formula (3.15) in [61]. Then T4 acts on the SU(r)/Z -theory to itself; and ST4S = r 4 1 will map the SU(r)-theory to itself. Note that Γ (4) = T, ST4S is generated by T, ST4S. In the case 0 x y of a spin manifold, we get the subgroup of SL(2, Z) generated by S and ST2S, which is the group a b Γ (2) = SL(2, Z) :2 c . 0 c d P | " * We will see this for K3 surfaces. Therefore if the S-duality conjecture holds, the partition function Γ Γ Z(SU(r); τ) = Z(SU(r); q) is a modular form with modular group 0(4) or 0(2) if M is a spin manifold. In [61, §4], Vafa-Witten checked the S-duality for the cases K3 surface and P2, and gave a prediction on a formula (5.38) of [61, §5]) for general type surfaces. For P2, Vafa-Witten used the mathematical results of Klyachko and Yoshioka, and for K3 surfaces, they predicted the formula from physics. In algebraic geometry the instanton invariants are the Euler characteristic of the moduli space of Gieseker or slope stable coherent sheaves on S. This corresponds to the case in [61] such that the COUNTING TWISTED SHEAVES AND S-DUALITY 3 obstruction sheaves vanish. It is worth mentioning that the blow-up formula of the Vafa-Witten invariants in this case was proved by Li-Qin in [42]. But to the author’s knowledge there are few theories or deﬁning invariants in algebraic geometry for the Langlands dual group SU(r)/Zr. In the rank 2 case, the Langlands dual group SU(2)/Z2 = SO(3). There exist some theories for the SO(3)-Donaldson invariants for the surface S, see [35], [16], [51].

1.2. Twisted Vafa-Witten invariants. In differential geometry solutions of the Vafa-Witten equation on a projective surface S are given by polystable Higgs bundles on the surface S, see [55]. The moduli space of Higgs bundles has a partial compactification by Gieseker semistable Higgs pairs (E, φ) on S, where E is a torsion free coherent sheaf with rank rk 0, and φ Hom (E, E K ) is a section ą P S b S called a Higgs field. Tanaka and Thomas [55], [56] have developed a theory of Vafa-Witten invariants using the moduli space N of Gieseker semi-stable Higgs pairs (E, φ) on S with topological data (rk = N K rank, c1(E), c2(E)). They actually defined the Vafa-Witten invariants using the moduli space L of Higgs pairs with fixed determinant L Pic(S) and trace-free φ. P N K For the gauge group SU(r), as in the case of [55], the structure group for the moduli space L L is SLr(κ). The Langlands dual group of SLr(κ) is SLr(κ) = PGLr(κ). Thus the corresponding moduli space, for the gauge group SU(r)/Zr, should be the moduli space of PGLr(κ)-Higgs bundles or sheaves. We consider µr-gerbes on S, where µr is the cyclic group of order r. From [45], for a µ -gerbe S S over a surface S such that the corresponding Brauer class is nontrivial, the moduli r Ñ stack of semistable S-twisted sheaves, in some sense, can be taken as a cover over the moduli space of semistable generalized Azumaya algebras on S, which in turn, is isomorphic to the the moduli space of PGLr(κ)-bundles or sheaves on S. This statement is generalized to the twisted Higgs sheaves in [27]. Therefore it is reasonable to propose that the Vafa-Witten invariants for the µr-gerbes on S will give the mathematical invariants for the Langlands dual group SU(r)/Zr. Let p : S S be a µ -gerbe, and fix a polarization O (1). The gerbe S is a surface DM stack, and Ñ r S all the equivalence classes of µr-gerbes over S are classified by the second étale cohomology group 2 H (S, µr). We have the Vafa-Witten invariants studied in [24] by the moduli space of stable Higgs sheaves on S. But this is too broad for the S-duality conjecture, and we then restrict to a subcategory of sheaves on S, called the category of S-twisted sheaves in [43]. We believe that this is the right category to define twisted Vafa-Witten invariants and check the S-duality. For a µ -gerbe S S over a surface S and let IS be the inertia stack, the S-twisted sheaves were r Ñ defined in [43], and we will review it in §3.1. Roughly speaking the twisted sheaf is a sheaf with the transition function on an open cover of S is gerbe cocycle [S] H2(S, µ ).A µ -gerbe twisted sheaf is P r r always given by a character morphism χ : µ G , and it can be understood as a G -gerbe twisted r Ñ m m sheaf. From the exact sequence (¨)r 1 µ O˚ O˚ 1 Ñ r Ñ S ÝÑ S Ñ and the long exact sequence for cohomology ϕ (1.2.1) H1(S, O˚) H2(S, µ ) H2(S, O˚) ¨¨¨Ñ S Ñ r ÝÑ S Ñ¨¨¨ A µ -gerbe S S corresponds to a G -gerbe [ϕ([S])]. A quasi-coherent sheaf E on the G -gerbe r Ñ m m [ϕ([S])] has a canonical decomposition

E = Ei i à where E is the eigensheaf on which the structure group G acts by λ f = λi f .A G -gerbe twisted i m ¨ m sheaf is a quasi-coherent sheaf E such that the structure group (stabilizer group) action G E E m ˆ Ñ is given by scalar multiplication, i.e., E = E1. g g For a S-twisted sheaf E, the geometric Hilbert polynomial is deﬁned as PE(m) = χ (E(m)), where g χ (E)=[IS : S] deg(Ch(E) TdS) ¨ 4 YUNFENG JIANG and [IS : S] is the degree of the inertia stack over the stack S. Then we can write down

d i g m P (m) = α (E) . E i i! = iÿ0 g The reduced geometric Hilbert polynomial for pure sheaves, which is denoted by pE; is the monic g P polynomial with rational coefficients E . Let E be a pure S-twisted coherent sheaf, it is semistable αd(E) g g if for every proper twisted subsheaf F E we have p p and it is stable if the same is true with a Ă F ď E strict inequality. Then fixing a geometric Hilbert polynomial P, the moduli stack of S-semistable M Mss,tw coherent sheaves := S/κ(P) on S is constructed in [43]. If the stability and semistability coincide, the coarse moduli space M is a projective scheme. The S-twisted Higgs sheaves can be similarly defined. The twisted Higgs pair (E, φ) is semistable g g if for every proper φ-invariant subsheaf F E we have p p . Let N tw := N s,tw (P) be the moduli Ă F ď E S/κ stack of stable S-twisted Higgs pairs on S with geometric Hilbert polynomial P. Let X := Tot(KS) be the canonical line bundle of S, then X is a smooth Calabi-Yau threefold DM 2 stack. The DM stack X X := Tot(KS) is also a µr-gerbe, and has the same class in H (S, µr) = 2 Ñ H (X, µr). By spectral theory again, the category of S-twisted Higgs pairs on S is equivalent to the category of X-twisted torsion sheaves E on X supporting on S X. Let π : X S be the projection. φ Ă Ñ One can take a projectivization X = Proj(KS OS), and consider the moduli space of X-twisted ‘ stable torsion sheaves on X with geometric Hilbert polynomial P. The open part that is supported on the zero section S is isomorphic to the moduli stack of stable S-twisted Higgs pairs N tw on S with geometric Hilbert polynomial P. N K,tw We still restrict to the moduli stack L of stable X-twisted Higgs pairs (E, φ) with fixed determinant L and trace-free on φ. There is also a symmetric perfect obstruction theory on the moduli stack N K,tw. Hence a virtual cycle [N K,tw]vir H (N K,tw). The moduli stack N K,tw is non- L L P 0 L L compact, but admits a Gm-action scaling the Higgs field φ. The invariants are defined by using virtual N K,tw Gm vir vir localization in [14]. Let [( L ) ] be the induced virtual cycle on the fixed loci, and N be the virtual normal bundle. For the DM stack S, orbifold Grothendieck-Riemann-Roch theorem implies that fixing a geometric Hilbert polynomial P is the same as fixing data α =(r = rk, L, c ) H˚(S, Q). 2 P Definition 1.3. (Definition 4.4) We define 1 (1.2.2) VWtw(S) := . α K vir [(N ,tw)Gm ]vir e(N ) ż L We call it the big twisted Vafa-Witten invariant for the gauge group SU(r)/Zr. N K,tw On the moduli stack L , the Behrend function K,tw νN : N Z L Ñ is defined in [6]. Definition 1.4. (Definition 4.10) We define tw N K,tw vwα (S) = χ( L , νN ) as the weighted Euler characteristic. We call it the small twisted Vafa-Witten invariant for the gauge group SU(r)/Zr.

tw We will see later that the small Vafa-Witten invariants vwα (S) take important role in the calculations. If r = 1, then any µr-gerbe S = S, and we cover the Vafa-Witten invariants VWα(S) and vwα(S) defined in [55]. N K,tw G G The moduli space L admits a m-action induced by the m-action on the total space X of the G N K,tw canonical line bundle KS. There are two type of m-fixed loci on L . The first one corresponds COUNTING TWISTED SHEAVES AND S-DUALITY 5

to the Gm-fixed S-twisted Higgs pairs (E, φ) such that the Higgs fields φ = 0. This fixed locus is Ms,tw just the moduli stack S/κ(P) of stable S-twisted torsion free sheaves E on S. This is called the Instanton Branch as in [55]. The second type corresponds to Gm-fixed S-twisted Higgs pairs (E, φ) such that the Higgs fields φ 0. This case mostly happens when the surfaces S are general type, ‰ and this component is called the Monopole branch. See §4.3 for more details. It is interesting to do the calculations for µr-gerbes on surfaces for this branch.

1.3. S-duality conjecture-Global view. Our goal is to use the twisted Vafa-Witten invariants tw tw VW , vw as in Definitions 1.3 and 1.4 to study the SU(r)/Zr-Vafa-Witten invariants. These twisted invariants, up to now, are defined using the moduli stack of S-twisted stable sheaves or Higgs sheaves. We first need to generalize them to count strictly semistable S-twisted sheaves or Higgs sheaves. Let S S be a µ -gerbe over a surface S. The S-twisted sheaves on S, or the S-twisted Higgs Ñ r sheaves forms a category. There are stability conditions on it. Then we apply the technique of [31], [32] to count the semistable objects in this category. When applying to the category of coherent sheaves on a Calabi-Yau threefold, one gets the generalized Donaldson-Thomas invariants. We apply it to the category of S-twisted Higgs sheaves to get the generalized twisted Vafa-Witten invariants. tw We briefly review the construction and see §5 for more details. On the category Cohc (X) of S-twisted Higgs sheaves, the Hall algebra H(Atw) is an algebra over K(St /κ), the relative ss,tw Grothendieck ring of stacks. The moduli stack Nα (X) of Gieseker semistable X-twisted sheaves is an element in the Hall algebra H(Atw). Then Joyce [32] defined an element ǫ(α), called the virtually indecomposable object in H(Atw) for a class α K (X). Joyce proves that this element has the special P 0 form in the Hall algebra and apply the integration map (basically applying the Behrend function to tw get weighted Euler characteristic), and get the generalized Vafa-Witten invariants JSα (X). See §5.2 for more details. For α = (rk, L, c ) H˚(S, Q), the generalized twisted Vafa-Witten invariant vwtw 2 P is defined by

0 (1.3.1) vwtw(S) :=( 1)h (KS) JStw(X) Q. α ´ α P tw Similar to [56], the big generalized twisted Vafa-Witten invariants VWα (S) are defined by tw Conjecture 5.10, motivated by the wall crossing formula of Joyce-Song [31] for vwα (S). If tw semistablity coincides with stability, then the twisted Vafa-Witten invariants VWα (S) is the one in Definition 1.3. If the µ -gerbe S S is trivial, i.e., S =[S/µ ], then the twisted Vafa-Witten invariants VWtw, vwtw r Ñ r (stable ones or generalized ones) are the same as the Vafa-Witten invariants VW, vw defined in [55], [56]. Then Conjecture 5.10 is true for KS 0 and K3 surfaces. For other µr-gerbe S S, if the moduli N s,tw tw ă tw Ñ stack S/κ(α) is smooth, then VW = vw . Recall that a µ -gerbe S S is called essentially trivial if its class is in the image of the morphism r Ñ H1(S, O˚) H2(S, µ ). S Ñ r Thus an essentially trivial µr-gerbe S S is given by a line bundle L Pic(S).A µr-gerbe Ñ 2 ˚ P S S is called optimal if the order [S] H (S, O )tor in the cohomological Brauer group 1 Ñ 2 O˚ | | P Br (S) = H (S, S )tor is of order r. From the long exact sequence (1.2.1), an essentially trivial µr- gerbe S S has image zero in H2(S, O˚). Ñ S It should be interesting to study the Conjecture 5.10 for essentially trivial µr-gerbes or optimal tw tw µr-gerbes on S, and prove that VW = vw for essentially trivial µr-gerbes or optimal µr-gerbes over a K3 surface, and find their difference. In this paper we only use its first term and we use the difference to calculate the twisted Vafa-Witten invariants for K3 surfaces and prove the S-duality for prime ranks. It should be possible to generalize the work in [48] to µr-gerbes S on a K3 surface and prove VWtw(S) = vwtw(S); and a multiple cover formula for the generalized Vafa-Witten invariants of Toda [60]. 6 YUNFENG JIANG

Remark 1.5. Before we define the SU(r)/Zr-Vafa-Witten invariants, we make a remark that if in the µr-gerbe S S, r = 1, then S is just the surface S. Then the invariants VWtw(S), vwtw(S) we defined are just the Ñ Vafa-Witten invariants VW(S), vw(S) defined in [55], [56].

Now we define the SU(r)/Zr-Vafa-Witten invariants. Let S be a smooth projective surface. Definition 1.6. Fix an r Z , for any µ -gerbe p : S S corresponding to g H2(S, µ ), let L P ą0 r g Ñ P r P Pic(Sg) and let tw c Z L(Sg, q) := VW (Sg)q 2 r, (r,L,c2) c ÿ2 be the generating function of the twisted Vafa-Witten invariants. Let us fix a line bundle L Pic(S), and define for any essentially trivial µr-gerbe Sg S corresponding to P ˚ Ñ ˚ the line bundle Lg Pic(S), Lg := p L Lg; for all the other µr-gerbe Sg S, keep the same Lg = p L. P 2 b 1 ÑO˚ 2 Also for L Pic(S), let L H (S, µr) be the image under the morphism H (S, S ) H (S, µr). We defineP P Ñ 2πig¨L r Zr,L(S, SU(r)/Zr; q) := e Zr,Lg (Sg, q). 2 gPHÿ(S,µr) We call it the partition function of SU(r)/Zr-Vafa-Witten invariants. This is parallel to the physics conjecture in [39, (5.22)]. Conjecture 1.7. (Conjecture 5.14) For a smooth projective surface S, the partition function of SU(r)-Vafa- Witten invariants ( ( ) ) = ( ) c2 Zr,L S,SU r ; q VW(r,L,c2) S q c ÿ2 and the partition function of SU(r)/Zr-Vafa-Witten invariants Zr,L(S,SU(r)/Zr; q) satisfy the S-duality conjecture in Conjecture (1.1.2). 2 Our first result is for the projective plane P . Since K 2 0, there are no second component P ă (Monopole Branch) for the moduli space of semistable Higgs sheaves as in [55]. In this case the Vafa- (P2) = (P2) Witten invariants VW(2,O,c2) vw(2,0,c2) is just (up to a sign) the Euler characteristic of the ss 2 moduli space M (P ) of semistable torsion free sheaves. Let N 2 (2, c , c ) be the moduli space of 2,0,c2 P 1 2 stable vector bundles of rank 2, first Chern class c1 and second Chern class c2. Let 2 vb,P c2 Z (q) = χ(N 2 ( c c ))q c1 P 2, 1, 2 c ÿ2 be the partition function. 2 2 2 2 The µ2-gerbes on P are classified by H (P , µ2) = µ2. Thus one is the trivial µ2-gerbe [P /µ2], and the other is the nontrivial µ2-gerbe corresponding to the nontrivial line bundle O( 1) on 2 2 ´ P , it is P(2,2,2), the weighted projective stack. The µ2-gerbes on P are all essentially trivial, and the P(2,2,2)-twisted sheaves behave like the sheaves on P(2,2,2). Then in this case the (P2) = (P2) VW(2,O,c2) vw(2,0,c2) is just (up to a sign) the Euler characteristic of the moduli space Mss (P2) of semistable torsion free sheaves. From a result as in [15], we have that in this case 2,0,c2 the first Chern class c is always even. Let λ 0,1 index the component in the inertia stack 1 P t u IP(2,2,2) = P(2,2,2) P(2,2,2). Let N (2, c , c ) be the moduli space of stable vector bundles Y P(2,2,2) 1 2 of rank 2, first Chern class c1 and second Chern class c2. Let P vb, (2,2,2) c2 Z (q) = χ(NP (2, c , c ))q c1,λ (2,2,2) 1 2 c ÿ2 be the partition function. Then we have Theorem 1.8. (Theorem 5.19) We define

P2 1 ´2 vb,P(2,2,2) ´ 15 vb,P(2,2,2) Z (SU(2)/Z ; τ) := q Z (q) + q 4 Z (q) 0 2 2 ¨ ¨ 0,0 ¨ 0,1 COUNTING TWISTED SHEAVES AND S-DUALITY 7 and P2 1 ´6 vb,P(2,2,2) ´ 15 vb,P(2,2,2) Z (SU(2)/Z ; τ) := q Z (q) q 4 Z (q) . 1 2 2 ¨ 2,1 ´ ¨ 2,0 Write P2 ´2 vb,P2 P2 ´ 15 vb,P2 Z (SU(2); τ) = q Z (q); Z (SU(2); τ) = q 4 Z (q). 0 ¨ 0 1 ¨ 1 Then under the S-transformation τ 1 , we have: ÞÑ ´ τ 3 P2 1 ´ 3 τ 2 P2 Z SU(2); = 2 2 Z (SU(2)/Z ; τ) 0 ´ τ ˘ i 0 2 and 3 P2 1 ´ 3 τ 2 P2 Z SU(2); = 2 2 Z (SU(2)/Z ; τ) 1 ´ τ ˘ i 1 2 vb,P(2,2,2) The S-duality conjecture is proved based on the observation that Z0,0 (q) is the same as the 2 partition of the trivial µ2-gerbe [P /µ2], since the component of IP(2,2,2) corresponding to λ = 0 vb,P(2,2,2) vb,P(2,2,2) means that µ2-action is trivial; and Z0,1 (q) is the same as the partition function Z2,0 (q) from Theorem 5.16. The proof for the second transformation formula is similar.

1.4. S-duality conjecture-K3 surfaces. Let S be a K3 surface. The following result for K3 surfaces was proved in [56, Theorem 1.7] by calculating the invariants vwα(S) using Toda’s multiple cover formula [60]: d´ 1 r ´24 c d r 2πij (1.4.1) Z (S,SU(r); q) := VW (S)q 2 = q η e d q d2 . r,0 r,0,c2 r2 c = ÿ2 ÿd|r jÿ0 where 1 k η(q) = q 24 (1 q ) ´ kźą0 is the Dedekind eta function; and Zr,0(S,SU(r); q) is the generating series of rank r trivial determinant Vafa-Witten invariants. If r is a prime number, then

r´1 ´24 c 1 r r ´ 1 r 2πij 1 (1.4.2) VW (S)q 2 = q η(q ) 24 + q η e r q r , r,0,c2 r2 r c = ÿ2 jÿ0 which is the prediction formula in [61, §4.1].

1.4.1. Rank 2 S-transformation. Let us study in detail in the rank two case. Let

Z(SU(2); q) := Z2,0(S,SU(2); q); Z(SU(2)/Z2; q) := Z2,0(S,SU(2)/Z2; q) after ﬁxing the rank 2 = r, trivial determinant det(E) = O. First we write down Tanaka-Thomas’s partition function as

c2 (1.4.3) Z(SU(2); q) := VW2,0,c2 (S)q c ÿ2 ´24 ´24 1 ´ 1 1 1 = q2η(q2) 24 + q2 η q 2 + η q 2 4 2 ´ 1 1 1 1 = q2G(q2) + q2 G(q 2 ) + G( q 2 ) 4 2 ´ Here q = e2πiτ and τ is the parameter fo the upper half plane. We use Vafa-Witten’s notation in [61] and denote by G(q) := η(q)´24. 8 YUNFENG JIANG

Under the S-transformation τ 1 , from modular transformation properties [61, §4.1], we have ÞÑ ´ τ 1 ´ 1 G( q 2 ) τ 12G( q 2 ), ´ ÞÑ ´ 1 ´12 ´12 2 $G(q 2 ) 2 τ G(q ), ÞÑ ´ 1 &’G(q2) 212τ 12G(q 2 ). ÞÑ From [61, §4.1], ﬁrst we shift back q2%’, and under S transformation

1 ´ 1 1 ´ ´ 1 ´ 1 Z(SU(2); q) 212τ 12G(q 2 ) + 2 12τ 12G(q2) + τ 12G( q 2 ). ÞÑ 4 2 2 ´ Thus from (1.1.1), after modifying 211, and shift back q2 we get

1 1 1 (1.4.4) Z(SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) , 2 4 ¨ ¨ ´ which should be the partition function Z(SU(2)/Z2, q) for the K3 surface S. This is Formula (4.18) in Γ [61], and has modular properties for 0(2). 1.4.2. Proof of rank two formula. We prove that the twisted Vafa-Witten invariants VWtw (also vwtw) in Conjecture 5.10 for the µ2-gerbes on S give the formula (1.4.4). Thus we prove Conjecture 1.7 for K3 surfaces S in rank two. We have the following result: Theorem 1.9. (Theorem 6.20) Let S be a smooth projective K3 surface with Picard number ρ(S). Then

1 1 ρ(S)´ 1 Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 2 1 G( q 2 ) . 2 4 ¨ ¨ ´ Corollary 1.10. (Corollary 6.21) Let S be a smooth projective K3 surface with Picard number ρ(S) = 11. Then 1 1 1 Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) . 2 4 ¨ ¨ ´ It is fun to include the calculation by Maple of the series Z(S, SU(2); q) and Z(S,SU(2)/Z2; q) for a smooth projective K3 surface S with Picard number ρ(S) = 11. We have

1 1 1 1 (1.4.5) Z(S,SU(2); q) = q2G(q2) + q2 G(q 2 ) + G( q 2 ) 4 2 ´ 1 = + 30q2 + 3200q3 + 176337q4 + 5930496q5 4 + 143184800q6 + 2705114280q7 + O(q8) which is the formula of Tanaka-Thomas in [56, §5].

1 1 1 (1.4.6) Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) 2 4 ¨ ¨ ´ 1 3 5 = + 2096128q 2 + 50356230q2 + 679145472q 2 + 6714163200q3 4 7 9 + 53765683200q 2 + 369816109137q4 + 2250654556160q2 + 11 + 12443224375296q5 + 63258156057600q 2 + O(q6)

SU(2)/Z2 From this calculation, if we deﬁne vwα (S) as the Vafa-Witten invariants for the SU(2)/Z2 theory, we get that ( ) Z 1 vwSU 2 / 2(S) = 2,0 4 and SU(2)/Z2 1 22 11 vw 3 (S) = (2 2 ) = 2096128. 2, 2 2 ´ COUNTING TWISTED SHEAVES AND S-DUALITY 9

Note that 1 (222 211) = 221 210 = 2096128, and this invariant is purely given by optimal µ -gerbes 2 ´ ´ 2 on S. The method to prove Theorem 1.9 is the following. Let S be a smooth K3 surface with Picard number ρ(S), then 0 ρ(S) 20 over a field κ of character zero. Over positive characteristic p 0 ď ď ą field κ, the Picard number can be ρ(S) = 22, which in this case S is called a “supersingular K3”. 2 22 ρ(S) The cohomology H (S, µ2) = Z2 . Since the Picard number is ρ(S), there are totally 2 number of equivalent essentially trivial µ2-gerbes on S; and all the other µ2-gerbes on S are optimal, meaning that the order [S] = 2 in H2(S, O˚) . There are totally 222 2ρ(S) number of equivalent optimal | | S tor ´ µ2-gerbes on S. For the trivial µ -gerbe S S, the twisted Vafa-Witten invariants VWtw(S) = vwtw(S) are the 2 Ñ same as usual Vafa-Witten invariants for S in [56], see Proposition 6.8. For the non-trivial essentially trivial µ -gerbe S S, the twisted Vafa-Witten invariants VWtw(S) = vwtw(S) are calculated in 2 Ñ Proposition 6.12. They behave a bit different from the trivial gerbe case, based on one fact that a rank two vector bundle on S with non-trivial second Stiefel-Whitney class (first Chern class (modulo 2) ) can not split. Let S S be an optimal µ -gerbe over S. In this case we use the theory of Huybrechts and Stellari Ñ 2 [19] for the Hodge structure for twisted K3, and of Yoshioka [65] for the moduli stack of gerbe twisted sheaves. An important observation is that the index ind(S) of the gerbe S, which by definition the minimal rank r such that there exists a S-twisted locally free of rank r sheaf on generic scheme S, is actually the period per(S) = [S] = 2, i.e., the order of [S] in H2(S, O˚) . In the moduli stack of | | S tor rank 2 S-twisted sheaves or Higgs sheaves (E, φ), E must be stable. Thus studying the moduli stack of S-twisted sheaves in rank 2 is a non-commutative analogue of the Picard scheme in the sense that the S-twisted sheaves are essentially rank one right modules over an Azumaya algebra on S. Using this we show that the moduli stack of stable S-twisted rank 2 Higgs sheaves is a µ2-gerbe over the Yoshioka moduli space of [S] (asa Gm-gerbe)-twisted stable Higgs sheaves on S. Then we use the theory of Yoshioka to calculate the twisted Vafa-Witten invariants VWtw(S) = vwtw(S), see Corollary 6.19. All the calculations imply the result in Theorem 1.9.

1.4.3. Vafa-Witten’s S-duality conjecture. If S is a complex K3 surface, the result in Corollary 1.10 depends on the complex structure on the K3 surface S. Vafa-Witten’s prediction in [61] does not depend on the complex structure. We modify Conjecture 1.7 a bit for K3 surfaces in rank two and 1 prove that the modified formula Z (S,SU(2)/Z2; q) does not depend on the complex structure. In the paper [61, Page 55], Vafa-Witten actually sum over topological data which means that they sum all the first Chern classes g H2(S, Z)/2 H2(S, Z) such that g2 0, 2 mod 4. There are totally three P ¨ ” types of these data: g = 0, g2 0 mod 4 but nonzero, and g2 2 mod 4 which are called the zero, ” ” even non-zero and odd classes respectively. Let n0 = 1, neven, nodd be the number of zero classes g, even non-zero classes g and odd classes g respectively. Then 222 + 211 222 211 n = 1; n = ´ . even 2 ´ odd 2 There are two types of non-zero even classes g H2(S, Z)/2 H2(S, Z). The first one consists of P ¨ algebraic classes (i.e. (1,1)-classes), and the number is n1 = (2ρ(S) 1). The second one consists even ´ of non-algebraic classes, and the number is n2 = n (2ρ(S) 1). Since the Picard number even even ´ ´ ρ(S) 20, these numbers make sense. ď Let S S be the trivial µ -gerbe; and S S a nontrivial essentially trivial µ -gerbe on S 0 Ñ 2 ess Ñ 2 corresponding to a line bundle L Pic(S). We set Z (q) := Z O(S , q) and calculate the partition g P 0 2, 0 functions: 1 2 1 1 Z L (S , q) = q G(q 2 ) + G( q 2 ) 2, g ess 2 ´ and 1 1 1 1 Z (q) = q2G(q2) + q2 G(q 2 ) + G( q 2 ) . 0 4 2 ´ 10 YUNFENG JIANG

Let S S be an optimal µ -gerbe on S, the twisted Vafa-Witten invariants are summing over opt Ñ 2 second Chern classes c2 taking values in half integers. We set

2 tw 2c2 c2 Z2,O(Sopt, ( 1) q) := vwv (Sopt)( 1) q ´ ¨ c ´ ÿ2 and calculate 1 2 1 2 1 2 1 Z O(S , q) = q G(q 2 ); Z O(S , ( 1) q) = q G( q 2 ). 2, opt 2 2, opt ´ ¨ 2 ´ tw For any line bundle L Pic(S), the moduli stack N (Sopt) of Sopt-twisted stable Higgs sheaves P 2,L,c2 N tw with fixed determinant L is isomorphic to the moduli stack O 1 (Sopt) with trivial determinant. 2, ,c2 Therefore the partition function Z (S , q) or Z (S , ( 1)2 q) is the same as Z O(S , q) or 2,L opt 2,L opt ´ ¨ 2, opt Z O(S , ( 1)2 q). 2, opt ´ ¨ For the even classes g H2(S, Z)/2 H2(S, Z) such that g 0, we define P ¨ ‰ 2 Z (q) := Z O(S , q) + Z O(S , ( 1) q). even 2, opt 2, opt ´ ¨ For odd classes g H2(S, Z)/2 H2(S, Z), we define P ¨ 2πi 1 2 Z (q) := Z O(S , q)+( 1) 2 Z O(S , ( 1) q). odd 2, opt ´ ¨ 2, opt ´ ¨ Then we have: Theorem 1.11. (Theorem 6.30) Let S be a smooth complex K3 surface. Define 1 Z O(S,SU(2)/Z ; q) := Z (q) + n Z (q) + n Z (q). 2, 2 0 even ¨ even odd ¨ odd We call it the partition function of SU(2)/Z2-Vafa-Witten invariants, and we have:

1 1 1 1 Z (S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) . 2 4 ¨ ¨ ´ This proves the prediction of Vafa-Witten in [61, Formula (4.18)] for the gauge group SU(2)/Z2 and the S-duality conjecture (1.1.2), and (1.4.4). The formula is a direct calculation from the proof of Theorem 1.9, see §6.6. We also talk about the higher rank case for the K3 surface S. For the prime rank r, the result for the partition function of SU(r)/Zr-Vafa-Witten invariants and Vafa-Witten’s conjecture can be similarly obtained as in the rank 2 case, see Theorem 6.36, and Theorem 6.38. The nonprime rank r case is generalized in [29], where a twisted version of Toda’s multiple cover formula is essential. In [28], the authors prove the refined version of the S-duality conjecture of K3 surfaces using the K-theoretical refined Vafa-Witten invariants [59]. 1.5. Outline. This paper is outlined as follows. We review the basic materials of algebraic stacks, Deligne-Mumford stacks and gerbes in §2. We define the moduli of gerbe twisted semistable sheaves and semistable Higgs sheaves in §3. We study the perfect obstruction theory of the moduli stack of twisted Higgs sheaves in §4 and define the twisted Vafa-Witten invariants. We study the Joyce- Song twisted stable pairs in §5 and define generalized twisted Vafa-Witten invariants. We also define the SU(r)/Zr-Vafa-Witten invariants and conjecture that its partition function satisfies the S-duality conjecture. The case of P2 is proved in rank two. Finally in §6 we prove the S-duality conjecture for K3 surfaces for prime ranks. 1.6. Convention. We work over an algebraically closed field κ of character zero throughout of the paper. We denote by Gm the multiplicative group over κ. We use Roman letter E to represent a coherent sheaf on a projective DM stack or an étale gerbe S, and use curl latter E to represent the sheaves on the total space Tot(L) of a line bundle L over S. We reserve rk for the rank of the torsion free coherent sheaves E, and when checking the S-duality for SU(r)/Zr, r = rk. We keep the convention in [61] to use SU(r)/Zr as the Langlands dual group of SU(r). When discussing the S-duality the rank rk = r. COUNTING TWISTED SHEAVES AND S-DUALITY 11

Acknowledgments. Y. J. would like to thank Kai Behrend, Huai-Liang Chang, Amin Gholampour, Shui Guo, Martijn Kool, Yang Li, Song Sun, and Hsian-Hua Tseng for valuable discussions. Many thanks to Martijn Kool for his interest in the paper and valuable comments, and Richard Thomas for his nice comments of the paper and communication of Brauer classes, especially explaining the idea of Vafa-Witten of summing over Chern classes for PGL2-bundles on K3 surfaces. This work is partially supported by NSF DMS-1600997.

2. PRELIMINARIES ON ETALE´ GERBES AND STACKS In this section we review the basic materials of stacks. Our main reference is [40] and [54]. Since we mainly work on étale µr-gerbes over schemes and µr-gerbes are closely related to Gm-gerbes on a scheme, we recall the basic notion of algebraic stacks (Artin stacks) and also Deligne-Mumford (DM) stacks. Several interesting examples and basic knowledge were reviewed in [24]. 2.1. Algebraic stacks and Deligne-Mumford stacks. Let us always fix a Noetherian scheme R, and let (Sch/R) be the Grothendieck topology, either étale or fppf topology. The étale topology will give us DM stacks and fppf topology will give us algebraic stacks (Artin stacks). In later sections we take R to be the base field κ. Let us first fix notations. Let X be a functor and f : U U be a morphism of schemes and i i Ñ X X(U) an object. Denote by X := f ˚X and X = f ˚ X where f : U U U is the P |i i i|ij ij,i i ij,i i Û j Ñ i morphism and X X(U ). i P i Definition 2.1. A stack is a sheaf X of groupoid, i.e., a 2-functor (presheaf) that satisfies the following sheaf axioms. Let U U be a covering of U in the site (Sch/R). Then t i Ñ uiPI (1) (Gluing) If X andY aretwoobjects of X(U), and ϕ : X Y are morphisms such that ϕ = ϕ , i |i Ñ |i i|ij j|ij then there exists a morphism η : X Y such that η = ϕ . Ñ |i i (2) (Mono presheaf) If X and Y are two objects of X(U), and ϕ : X Y, ψ : X Y are morphisms such Ñ Ñ that ϕ = ψ , then ϕ = ψ. |i |i (3) (Gluing of objects) If Xi are objects of X(Ui) and ϕij : Xj ij Xi ij are morphisms satisfying the | Ñ | – cocycle condition ϕ ϕ = ϕ , then there exists an object X of X(U) ans ϕ : X X ij|ijk ˝ jk|ijk ik|ijk i |i ÝÑ i such that ϕ ϕ = ϕ . ji ˝ i|ij j|ij Definition 2.2. We take Sch/R to be the category of R-schemes with the étale topology. Let X be a stack. X is a DM stack if (1) The diagonal ∆X is representable, quasi-compact and separated. (2) There exists a scheme U (called atlas) and an étale (hence locally of finite type) and surjective morphism u : U X. Ñ A morphism f : X Y of DM stacks is called “representable” if every morphism g : S Y from Ñ Ñ a scheme S, the fibre product S Y is a scheme. In particular, any morphism from a scheme to ˆg,Y, f Y is representable. Definition 2.3. Let X be a smooth DM stack. The inertia stack IX associated to X is defined to be the fibre product: IX := X ∆ X ˆ ,XˆX where ∆ : X X X is the diagonal morphism. Ñ ˆ The objects in the category underlying IX is:

Ob(IX) = (x, g) x Ob(X), g AutX(x) . t | P P u Remark 2.4. (1) There exists a morphism q : IX X given by (x, g) x; Ñ ÞÑ (2) There exists a decomposition

IX = Hom(Rep(Bµr, X)), PN rğ 12 YUNFENG JIANG

hence IX can be decomposed into connected components, and

IX = Xi PI iğ where I is the index set. We have X = (x, id) x Ob(X), id AutX(X) = X. 0 t | P P u Deﬁnition 2.5. Let X be a stack over the site (Sch/R) with the fppf topology. Assume that

(1) The diagonal ∆X is representable, quasi-compact and separated. (2) There exists a scheme U (called atlas) and a smooth and surjective morphism u : U X in fppf topology. Ñ Then we call X an Artin stack. Example 1. A very interesting stack is given by the quotient stack X = [X/G], where X is a smooth scheme and G an algebraic group acting on X. The stack [X/G] classiﬁes isomorphic classes of principle G-bundles E over a scheme B [X/G] together with a G-equivariant morphism: Ñ f E / X

u B / [X/G]

The smooth atlas of X is X [X/G]. If G is a finite group, then X is a DM stack. All stacks in this paper Ñ locally is a quotient stack. Let I = (g) g G , where (g) is the conjugacy class. The inertia stack IX = X , and X = t | P u (g) (g) (g) [Mg/C(g)], where Mg is the g fixed locus of M and C(g) is the centralizer of g. Ů 2.2. Gerbes and essentially trivial gerbes. Gerbes are interesting algebraic stacks. Let us fix a base algebraic stack X. Later on we always consider X to be a smooth scheme. Definition 2.6. A gerbe X over X is a stack X such that the following is satisfied: (1) For any U X there exists a covering U1 U such that X(U1) . P 1 Ñ 1‰ H (2) For any U X and any s, s X(U), there exists a covering U U such that s U is isomorphic to 1 P P Ñ | s 1 . |U Definition 2.7. Let X be a gerbe over X. The sheaf associated to X, denoted by Sh(X) is the sheafification of the presheaf whose sections over U X are isomorphic classes of objects in the fiber category X(U). Ă In this paper we mainly work on A-gerbes for an abelian group scheme A over R. First let X be the classifying topos of X, i.e., the topos of sheaves on the site of X. Then we have: If X X is a gerbe and the inertia stack IX is an abelian group sheaf on X, then there existsr an ‚ Ñ abelian sheaf A on X such that π˚ A IX – on X, where π : X X is the morphism of topoi. Ñ r Definition 2.8. An A-gerbe X over X is a gerbe X X such that A IX in X, where A is a sheaf of r r r X X abelian group A on X. Ñ – r Let X be a scheme, the A-gerbes X over X are classified by the second étale cohomology group 2 H (X, A). In this paper we are interested in the group A = µr or A = Gm, i.e., the µr-gerbes on a scheme X; or the Gm-gerbes on X. It is not hard to see that a µr-gerbe X on a scheme X is a DM stack. We always let χ : A G be a character. Ñ m Definition 2.9. A µ -gerbe X X on a scheme X is (geometrically) essentially trivial if the class [X] r Ñ P H2(X, µ ) has trivial image in H2(X, G ) under the natural map H2(X, µ ) H2(X, G ). r m r Ñ m COUNTING TWISTED SHEAVES AND S-DUALITY 13

Consider the short exact sequence: (¨)r 1 µ G G 1 Ñ r ÝÑ m ÝÑ m Ñ and taking cohomology we have the exact sequence (2.2.1) H1(X, G ) H2(X, µ ) H2(X, G ) ¨¨¨Ñ m ÝÑ r ÝÑ m Ñ¨¨¨ Thus a µ -gerbe X X is essentially trivial if and only if it is given by the image class of a line bundle r Ñ L Pic(X) under the ﬁrst map in (2.2.1). Also in this case one can understand the µr-gerbe X X as P ˆ ˆ Ñ a quotient [Tot(L )/Gm], where Tot(L ) denote the total space of the line bundle L minus the zero section X, and Gm acts on the ﬁber by the r-th power. 2.3. Brauer group and optimal gerbes. In this section we talk about the Brauer group and optimal gerbes. For the basic knowledge of Brauer group, see [49]. Let X be a scheme, the Brauer group Br(X) parametrizes equivalence classes of sheaves of Azumaya algebras on X.

Definition 2.10. An Azumaya algebra on X is an associative (non-commutative) OX-algebra A which is locally isomorphic to a matrix algebra M (O ) for some r 0. r X ą Two Azumaya algebras A and A1 are called equivalent if there exist non-zero vector bundles E and E1 such that A End(E) and A1 End(E1) are isomorphic Azumaya algebras. The Brauer group b b Br(X) is the set of isomorphism classes of Azumaya algebras modulo the above equivalence relation. 1 2 O˚ The cohomological Brauer group Br (X) is the torsion part H (X, X)tor of the étale cohomology 2 O˚ H (X, X), which classifies the equivalence classes of Gm-gerbes on X. The following result is due to de Jong. Theorem 2.11. (de Jong [12]) If X is a quasi-compact separate scheme admitting an ample invertible sheaf, then Br(X) = Br1(X). Definition 2.12. Let X X be a µ -gerbe. The period per(X) is the order of [X] in Br(X). A µ -gerbe Ñ r r X X is called optimal if the period per(X) = r. Ñ The Brauer group also classifies Brauer-Severi varieties, and these are projective bundles P over X, see [1]. For us, if we have an optimal µ -gerbe X X over a scheme X, then from Theorem 2.11, r Ñ Br(X) = Br1(X). We let o([X]) be the image of [X] under the map o : H2(X, µ ) H2(X, O˚ ) r Ñ X 2 O˚ A which must lie in the torsion part of H (X, X). Any Azumaya algebra is locally free of constant 2 rank r . Two Azumaya algebras are isomorphic if they are isomorphic as OX-algebras. By Skolem- Noether theorem, Aut(Mr(κ)) = PGLr(κ) (acting by conjugation). Therefore, the set of isomorphism 2 1 classes of Azumaya algebras A of rank r is in bijection with the set H (X, PGLr). From the exact sequence 1 µ SL PGL 1 Ñ r ÝÑ r ÝÑ r Ñ we have a morphism by taking cohomology δ1 : H1(X, PGL (O )) H2(X, µ ). We set δ := o δ1. r X Ñ r ˝ Then the optimal µ -gerbe [X] also defines a class [P] H1(X, PGL ) such that δ1[P]=[X]. This r P r projective bundle P X is the Brauer-Severi variety. Ñ Remark 2.13. If a µ -gerbe X X is essentially trivial, then its class [X] is zero in H2(X, G ), therefore also r Ñ m defines a zero class [P] H1(X, PGL ). In this case the Brauer-Severi variety P is just X. P r 3. MODULI SPACE OF TWISTED SHEAVES AND TWISTED HIGGS SHEAVES ON SURFACES In this section we recall the notion of µ -gerbe X-twisted sheaves on X X, and define its stability r Ñ conditions and moduli stacks. We mainly follow the arguments in [43]. We also generalized the µ -gerbe X-twisted sheaves on X X to twisted Higgs sheaves and define their moduli stack. r Ñ For sheaves on algebraic stacks, see [54, 06TF]. The category of gerbe X-twisted sheaves on the stack X is naturally a subcategory of the category of sheaves on X. 14 YUNFENG JIANG

3.1. Etale´ gerbe twisted sheaves. Let us fix A to be an abelian group scheme over a smooth scheme X. We follow [43], [10] for the notion of twisted sheaves. Let π : X X be a A-gerbe and we denote Ñ by [X] to be the class in H2(X, A). Let E be a sheaf on X, then one has a sheaf π˚E on X by the topoi morphism π : X X. Therefore there is a natural right group action Ñ r µ : E IX E, r ˆ Ñ see [43, Lemma 2.1.1.8]. The following result is from [43, Lemma 2.1.1.13]. Proposition 3.1. Let X X be an A-gerbe and E is a sheaf on X such that the inertia action E IX E is Ñ ˆ Ñ trivial, then E is naturally the pullback of a unique coherent sheaf on X up to isomorphism. The A-gerbe X-twisted coherent sheaves are defined as follows. Let χ : A G be the character Ñ m morphism. Let E be a coherent OX-module, the module action m : Gm E E yields an associated ˆ Ñ right action m1 : E G E by m1(s, ϕ) = m(ϕ´1, s). ˆ m Ñ Definition 3.2. A d-fold χ-twisted sheaf on X is a coherent OX-module E such that the natural action µ : E A E given by the A-gerbe structure makes the diagram ˆ Ñ E A / E ˆ χd id m1 E G / E ˆ m commutes, where χd(s) = χ(s)d. An 1-fold twisted coherent sheaf will be called a twisted sheaf. Remark 3.3. The gerbe X-twisted sheaves naturally form a fibered category of the classifying topos X, and can be viewed as a fibred category over X via the natural map X X of topoi. This fibred category of X-twisted Ñ sheaves is naturally a stack over X. r r Remark 3.4. In general, in [57], for a A-gerbe X X, there is a disconnected stack X (finite copies of the Ñ A-gerbe X) and a Gm-gerbe c (called the B-field) on X such that the category of coherent sheaves on X is equivalent to the category of gerbe c-twisted coherent sheaves on X. The twisted sheavesp of Lieblich are only on one component. Since we don’t need this in this paper,p we fix to Lieblich’s definition and moduli on twisted sheaves. p Example 2. Let A = G , and χ = id. Then let U be an open covering of X. A X-twisted sheaf on X is m t iu given by:

(1) a sheaf of modules Ei on each Ui; – – (2) for each i and j an isomorphism of modules g : E E such that on U ,E E ij j|Uij ÝÑ i|Uij ijk k|Uijk ÝÑ k|Uijk is equal to the multiplication by the scalar a G (U ) giving the 2-cocycle [X] H2(X, G ). P m ijk P m We are mainly interested in the A = µr-gerbe (or A = Gm-gerbe) twisted sheaves. Let χ : µ (or G ) G r m Ñ m be the natural inclusion of a subsheaf, a χ-twisted sheaf on X is just called an X-twisted sheaf. We list the following obvious result for reference: Proposition 3.5. A G -gerbe X X is an Artin stack locally of finite presented over X. A µ -gerbe X X m Ñ r Ñ is a DM stack. Definition 3.6. Let X X bea µ -gerbe over X. The index ind(X) is the minimal rank of a locally free Ñ r X-twisted sheaf over the generic scheme of X. Let us recall the period of a µ -gerbe X X in Definition 2.12. From [43, §2.2.5], for any scheme X, r Ñ one has per(X) ind(X). When X is a smooth surface, from a theorem of Je Jong, we have per(X) = | ind(X). Thus over smooth surfaces X, the rank of any locally free X-twisted sheaf is divisible by ind(X). The studying of moduli of twisted sheaves of rank r on an optimal µ -gerbe X X is a r Ñ COUNTING TWISTED SHEAVES AND S-DUALITY 15 non-commutative analogue of the Picard scheme, in the sense that such sheaves are essentially rank one right modules over an Azumaya algebra on X. 3.2. Twisted stability and the moduli stack. We review the twisted stability of X-twisted sheaves and construction of moduli spaces. 3.2.1. Twisted stability of Lieblich. We first define the support of twisted sheaves. Definition 3.7. Let X X bea µ -gerbe over X and E a X-twisted sheaf. The support supp(E) of E is the Ñ r closed substack of X defined by the kernel of the map OX EndX(E), which is a quasi-coherent sheaf of ideals. The schematic support of E is the scheme-theoretic imageÑ in X of the support of E. The support of E is the preimage of the schematic support. Then as in [43] Lieblich defined the torsion subsheaves of a stack X. One can define a torsion filtration on a twisted sheaf for any algebraic stack. We only use his definition and more details can be found in [43]. There exists a unique maximal coherent torsion subsheaf of E. The maximal torsion subsheaf of E is called the torsion subsheaf of E and is denoted by T(E). A coherent sheaf E is pure if T(E) = 0. Some properties of pure sheaves on stacks X can be found in [43, §2.2]. We define the stability condition of twisted sheaves and talk about the moduli spaces. We will define the moduli stack of semistable twisted sheaves on the µr-gerbe X. In order to fix topological invariants, let us take the Chow group A˚(X) with Q-coefficient as in [62]. Since X is a smooth DM stack, A˚(X) A˚(X). For – DM stacks, the better way to count the Hilbert polynomial is to use the Grothendieck K-group K0(X) of the abelian category of coherent sheaves. There exists an orbifold Chern character morphism: – K (X) H˚ (X) = H˚(IX) 0 ÝÑ CR given by E Ch(E) ÞÑ which is an isomorphism as Q-vector spaces. Let Td(TX) be the orbifold Todd class, then the orbifold Grothendieck-Riemann-Roch theorem says: Ă Ă (3.2.1) χ(X, E) = Ch(E) Td(TX) ¨ żIX see for instance [11]. Ă Ă We recall the definition of geometric stability conditions of Lieblich in [43]. Definition 3.8. The geometric Euler characteristic of a class α K (X) is defined as: P 0 g χ (α) :=[IX : X] deg(Ch(α) TdX). ¨ ¨ Let OX(1) be a polarization of X. Then the geometric Hilbert polynomial of α is: g m P (m) = χg(α O (m)). ÞÑ α b X Following [21, §1.2], for any E with class α, the Hilbert polynomial can be written down as:

dim E i g m P (m) = α (E) . E i i! iÿ=0 Note that here αi(E) do not need to be integers, since X is a DM stack. If E has dimension d, then the geometric rank is: α (E) rk(E) := d αd(OX) and the geometric degree of E is given by:

deg(E) = α (E) rk(E) α (OX). d´1 ´ ¨ d´1 Deﬁnition 3.9. Let E be a coherent sheaf on X of dimension zero, the length of E is deﬁned as: ℓ(E) = χg(E). 16 YUNFENG JIANG

Definition 3.10. Let X X bea µr-gerbe over a smooth scheme X. An X-twisted sheaf E of dimension d is semistable (resp. stable) ifÑ any subsheaf F E Ă g g α (E) P (resp. )α (F) P . d ¨ F ď ă d ¨ E g g P A semistable coherent X-twisted sheaf E must be pure. So let p = E , a gerbe X-twisted sheaf E αd(E) g g E is semistable if and only if it is pure and for any F E, p p . Ă F ď E Definition 3.11. The slope of a coherent X-twisted sheaf E of dimension d is defined by: deg(E) µ(E) := . rk(E)

Deﬁnition 3.12. Let X X be a µr-gerbe over a smooth scheme X. An X-twisted coherent sheaf E of dimension d is µ-semistableÑ (resp. µ-stable) if E is pure and for any subsheaf F E Ă µ(F) (resp. )µ(E). ď ă Lieblich [43] deﬁned the moduli stack of semistable (resp. stable) X-twisted sheaves on X using the above stability.

3.2.2. The moduli stack of X-twisted semistable sheaves. We use the geometric Hilbert polynomial to define the moduli stack of gerbe X-twisted sheaves studied in [43]. First we have: Proposition 3.13. ( [43, Corollary 2.3.2.11]) Let X X bea µ -gerbe over a smooth scheme X/R, and E is Ñ r an R-flat family of coherent X-twisted sheaves. The locus of µ-semistable (resp. semistable), (resp. geometrically µ-semistable or stable) fibers of E is open in R. Let us fix a Hilbert polynomial P Q[m], which is equivalent to fixing a K-group class c K (X). P P 0 From Proposition 3.13, there is an algebraic stack (3.2.2) Mss,tw : Sch/κ (Groupoids) X/κ Ñ of semistable twisted sheaves of fixed rank rk and Hilbert polynomial P. This stack contains an open Ms,tw substack X/κ of geometrically stable points. Let PicX/κ be the Picard stack of X parametrizing invertible sheaves on X. There is a determinant 1-morphism: Mtw (rk) Pic X/κ Ñ X/κ Mtw where X/κ(rk) is the stack of X-twisted coherent sheaves of rank rk (locally of finite type Artin stack). The stack of semistable X-twisted sheaves of rank rk, determinant L Pic and geometric P X/κ Hilbert polynomial P is defined by: Mss,tw(rk, L, P) := Mss,tw(rk, P) κ X/κ X/κ ˆPicX/κ,ϕL where ϕ : κ Pic is a morphism given by L. L Ñ X/κ For any algebraic stack M, and suppose that IM M is fppf, then the big étale sheaf Sh(M) Ñ associated to M is an algebraic space and M Sh(M) is a coarse moduli space, see [40], [54]. Ñ Therefore we have: Proposition 3.14. ([43, Proposition 2.3.3.4]) The stack Ms,tw (rk, L, P) Sh(Ms,tw (rk, L, P)) X/κ Ñ X/κ is a µr-gerbe over the algebraic space of finite type over κ. Next in the end of this section we collect some results for the moduli stacks for essentially trivial gerbes. Recall that from Definition 2.9, the µ -gerbe X X is essentially trivial if it is in the kernel of r Ñ the morphisms H1(X, G ) H2(X, µ ) H2(X, G ). m Ñ r Ñ m COUNTING TWISTED SHEAVES AND S-DUALITY 17

Such a µ -gerbe X can be given by a line bundle L X, i.e., X = [Tot(Lˆ)/G ] with the action r Ñ m λ(x, e)=(x, λr e). And also there exists an exact sequence: ¨ (3.2.3) 0 Pic(X)/r Pic(X) H2(X, µ ) Br(X)[r] 0. Ñ ¨ ÝÑ r ÝÑ Ñ We have the following result as in [43, Theorem 2.3.4.5]: Proposition 3.15. Suppose that X X is an essentially trivial µ -gerbe, then there exists a line bundle Ñ r γ Pic(X) 1 Z, such that the moduli stack Mss,tw (rk, L, P) is isomorphic to the stack Mss,γ (rk, L(rγ), P) P b r X/κ X/κ of γ-twisted semistable sheaves on X of rank rk, determinant L(rγ) and Hilbert polynomial P. The twisted stability on X is from [46], and we recall it here. Let L Pic(X) Q be a line bundle, P b and H = OX(1) is a polarization. A torsion-free sheaf E on X is said to be L-twisted H-Gieseker semistable for L if and only if for F E, Ă χ(F L Hm) χ(E L Hm) b b b b rk(F) ď rk(E) for m 0. ąą 3.3. Moduli of X-twisted sheaves for optimal gerbes. In this section we review the case of optimal µ -gerbes X X, i.e., for the class [X] H2(X, µ ), its image o([X]) in H2(X, O˚ ) has order r. We r Ñ P r X always assume that X is a smooth projective surface. For a smooth surface X, and a µ -gerbe X X r Ñ over X, fixing the geometric Hilbert polynomial P of torsion free sheaves E is the same as fixing the data (rk, L, c ) H˚(X). Our goal is to show that the moduli stack Ms,tw (rk, L, c ) is a µ -gerbe over 2 P X/κ 2 r the moduli stack of twisted semistable sheaves on X. From §2.3, the Brauer-Severi variety P X associated with the gerbe is a projective bundle over X r´1 Ñ with fiber P . From Definition 3.6, and the paragraph following on, the index ind(X) of a µr-gerbe X X is the minimal rank that there exists a locally free X-twisted sheaf over the generic scheme X. Ñ For the surface X, ind(X) = per(X) = r. Let ω(P) := δ1(P) H2(X, µ ). If there is a vector bundle P r E X such that P = P(E), then Ñ ω(P) = c1(E) mod r. For such a projective bundle P, there exists a vector bundle G on P which is given by the Euler sequence: 0 O G T /X 0 Ñ P ÝÑ ÝÑ P Ñ and G is a non-trivial extension of TP/X by OP. 3.3.1. Yoshioka’s moduli of semistable twisted sheaves on X. In [65], Yoshioka studied the moduli of twisted semistable sheaves on the Brauer-Severi variety P X, associated to the optimal µ -gerbe Ñ r [X] H2(X, µ ) such that under the morphism P r H2(X, µ ) H2(X, O˚ ), r ÝÑ X we have [X] α H2(X, O˚ ) [r] = H1(X, PGL ). ÞÑ P X tor r Yoshioka [65] defined the subcategory Coh(X; P) Coh(P) of coherent sheaves on P to be the Ă subcategory of sheaves E Coh(X; P) P if and only if E p˚(E ) O (λ ) |Pi – i b Pi i where U is an open covering of X, P = U Pr´1, E Coh(U ); and there exists an equivalence: t iu i i ˆ i P i – Coh(X; P) Coh(X, α) ÝÑ by _ E p˚(E L ). ÞÑ b 18 YUNFENG JIANG

Here α = o([X]) H2(X, O˚ ) is the image of the µ -gerbe [X]. The line bundle L Pic(P) is the P X tor r P line bundle with the property that

L ´1 = O ´1 ( 1), |p (x) p (x) ´ and Coh(X, α) is the category of α-twisted coherent sheaves on X, see Example 2. We call E P Coh(X; P) a P-sheaf. A P-sheaf E is of dimension d if p˚E is of dimension d on X. Yoshioka defined the Hilbert polynomial d G _ G m + i P (m) = χ(p˚(G E)(m)) = α (E) . E b i ¨ i = iÿ0 Definition 3.16. The P-sheaf E of dimension d is G-twisted semistable (with respect to OX(1)) if E is pure and for any proper subsheaf F E Ă _ _ χ(p˚(G F)(m)) χ(p˚(G E)(m)) b b G ď G αd (F) αd (E) for m 0. ąą Then we have the following result for the moduli spaces: P,G Theorem 3.17. ([65, Theorem 2.1]) There exists a coarse moduli space MX/κ(rk, P) is S-equivalence classes P,G of G-twisted semi-stable P-sheaves E with G-twisted Hilbert polynomial P, and moreover MX/κ(rk, P) is a projective scheme. 3.3.2. Artin-de-Jong’s smooth surface Y over X of degree ind(X). The Brauer-Severi variety P X 1 Ñ represents the class [X] H (X, PGLr). From §2.3, this projective bundle P corresponds to an A P MX Azumaya algebra P over X. From [45], one can define the moduli stack rk of generalized Azumaya algebras on X X, and the moduli stack of X-twisted sheaves Mtw (rk) is a cover over Ñ X/κ MX rk. The construction and the generalization to Higgs version of Azumaya algebras can be found in [45] and [27]. Thus any P-sheaf E Coh(X; P) from Yoshioka corresponds to a module E over the corresponding P Azumaya algebra AP and in [53] Simpson defined the stability for the underlying sheaf E by taking the Hilbert polynomial χ(X, E(m)). From [43, §2.3.4], [63], the stability defined by it is the same as the G-twisted stability in Definition 3.16. In [2], for the optimal µ -gerbe X X with [X] H2(X, O˚ ) , Artin-de-Jong constructed a r Ñ P X tor smooth surface Y and a morphism ϕ : Y X Ñ of degree r. We only recall the idea and more details can be found in [2]. The surface Y ã A L = M (O ) L Ñ P b r X b is cut of of the “characteristic polynomial” of a general section of A L for L a sufficiently ample P b invertible sheaf on X. In more detail, for any invertible sheaf L, the reduced norm yields an algebraic morphism A L Sym‚ L P b Ñ with image the polynomial sections of degree r. The zero locus of such a polynomial function on L_ gives the finite covering Y X of degree r, and Ñ Y / X ❄❄ ⑧? ❄❄ ⑧⑧ ❄❄ ⑧⑧ ❄❄ ⑧ ⑧⑧ X the map factors through the gerbe X. From [43, Proposition 3.2.2.6], COUNTING TWISTED SHEAVES AND S-DUALITY 19

Proposition 3.18. Let X X be an optimal gerbe over a surface X such that ind(X) = per(X) = r. Then there exists a locally free XÑ-twisted sheaf of rank r, and there exists a finite flat surjection of smooth surfaces ϕ : Y X Ñ of degree r such that (1) there exists an invertible X Y-twisted sheaf. ˆX (2) for every very ample invertible sheaf OX(1) on X, a general member has smooth preimage in Y. From this proposition, and Lemma 3.2.3.2, Lemma 3.2.2.3 in [43], Lieblich showed that any µ- semistable twisted sheaf E on S, the pullback ϕ˚E is µ-semistable. We hope that the moduli stack Mss,tw(rk, P) on X of X-twisted semistable sheaves on X is isomorphic to the moduli stack of X Y- X/κ ˆX twisted sheaves on Y. We don’t need this in this paper. We conclude: Ms,tw Theorem 3.19. The moduli stack X/κ(rk, L, P) of stable X-twisted sheaves on X is a µr-gerbe over the P,G moduli stack MX/κ(rk, L, P) of the G-twisted semistable sheaves on X. Proof. From Proposition 3.14, the stack Ms,tw (rk, L, P) Sh(Ms,tw (rk, L, P)) X/κ Ñ X/κ Ms,tw is a µr-gerbe over the algebraic space of finite type over κ. Here Sh( X/κ(rk, L, P)) is the big étale Ms,tw sheaf associated to the algebraic stack X/κ(rk, L, P). We only need to show that this algebraic space P,G is actually MX/κ(rk, L, P) of the G-twisted semistable sheaves on X. Since from [57], the category of coherent sheaves on X is isomorphic to G -gerbe twisted sheaves on a rigidified inertia stack X = X m \ X X. The twisted sheaves are exactly corresponding to the G -gerbe twisted sheaves on the \¨¨¨\ m first component in X, and this Gm-gerbe is the Brauer class of the gerbe X. Thus the Gm-gerbep twisted Ms,tw sheaves are actually the G-twisted semistable sheaf on X of Yoshioka. Therefore Sh( X/κ(rk, L, P)) P,G p is just MX/κ(rk, L, P). 3.4. Twisted Higgs sheaves and the moduli stack. From this section we work on a smooth projective surface S, and let S S be a µ -gerbe on a smooth surface S. In this section we define S-twisted Ñ r Higgs sheaves on a surface S. Although all arguments in this section work for higher dimensional projective schemes, we restrict to surfaces throughout. Let us fix a polarization O (1) on S for the µ -gerbe S S. Let L Pic(S) be a line bundle, S r Ñ P recall that a Higgs sheaf on S is a pair (E, φ), where E is a coherent sheaf on S and φ : E E K is Ñ b S OX-linear map called the “Higgs field”, see [55]. We consider the Higgs pairs on the µr-gerbe S. Let us fix a line bundle L Pic(S). Later on we are interested in L = KS, the canonical line bundle KS. P Definition 3.20. An L-Higgs sheaf on S is a pair (E, φ), where E is a coherent sheaf on S, and φ : E E L Ñ b a OS-linear morphism. Since S is a surface DM stack, from [24, Proposition 2.18], there exists an equivalence:

HiggL(S) Cohc(Tot(L)), – where we denote by Tot(L) the total space of the line bundle L. We denote by X := Tot(KS), the canonical line bundle of KS. This X is a Calabi-Yau threefold stack. Here Cohc(Tot(L)) is the abelian category of compactly supported sheaves on Tot(L). Let us always ﬁx the following diagram: p (3.4.1) X / X

π π p S / S where X = Tot(KS) is the total space of KS, which is the coarse moduli space of X. Proposition 3.21. The DM stack X X is also a µ -gerbe, and the class [X] H2(X, µ ) H2(S, µ ). Ñ r P r – r 20 YUNFENG JIANG

Proof. This can be seen étale locally since X is an affine bundle over S. We have H2(X, µ ) H2(S, µ ) r – r and X X is the µ -gerbe corresponding to the same class [S] H2(S, µ ). Ñ r P r Definition 3.22. An S-twisted L-Higgs sheaf on S is a pair (E, φ),where E isa S-twisted coherent sheaf on S as in Definition 3.2, and φ : E E L a OS-linear morphism such that the following diagram Ñ b φ E µ / E / E L ˆ r b χ id id m1 φ E G / E / E L ˆ m b commutes. Here is a result generalizing [24, Proposition 2.18] and [55, Proposition 2.2]. Proposition 3.23. ([24, Proposition 2.18]) There exists an abelian category Higgtw (S) of S-twisted Higgs KS pairs on S and an equivalence: „ (3.4.2) Higgtw (S) Cohtw(X) KS ÝÑ c tw where Cohc (X) is the category of compactly supported X-twisted coherent sheaves on X. Proof. First [24, Proposition 2.18] gives an equivalence:

Higg (S) Cohc(Tot(L)), KS – ˚ where if we let η be the tautological section of π KS on X, which is linear on the ﬁbers and cuts out S X. Then Ă O ´i i π˚( X) = KS η , iě0 ¨ à and the category of coherent OX-modules is equivalent to the category of coherent π˚(OX)-modules on S. Any π˚(OX)-coherent module π˚E corresponds to a coherent OS-module E, together with a OS-linear morphism π˚η E K´1 E. b S ÝÑ Thus a Higgs pair (E, φ) for φ = π˚η : E E KS and vice versa. The construction is compatible Ñ b with the twisted condition by the gerbe S S and X X, and the gerbe X-twisted sheaf E Ñ Ñ corresponds to m E µ / E ˆ r χ id 1 E G m / E ˆ m and the X-twisted sheaf actions χ, m, m1 correspond to the S-twisting:

m E = π˚E µ / E ˆ r χ id m1 E = π˚E Gm / E ˆ together with a Higgs ﬁeld φ : E E KS. Ñ b We deﬁne twisted stability for Higgs pairs. COUNTING TWISTED SHEAVES AND S-DUALITY 21

Deﬁnition 3.24. Let (E, φ) be a torsion free S-twisted Higgs sheaf on S. Then we say (E, φ) is Gieseker semi(stable) if for any subsheaf (F, φ1) (E, φ) we have Ă g g p (m) ( )p (m) F ď ă E for m 0. ąą For a torsion free S-twisted Higgs sheaf (E, φ) on S, we still deﬁne

degg(E) = r c (E) c (O (1)). ¨ 1 ¨ 1 S żS g and µ(E) = deg (E) . Then (E, φ) is µ-semi(stable) if for any subsheaf (F, φ1) (E, φ) we have rk(E) Ă µ(F) ( )µ(E). ď ă N ss,tw N ss,tw Let := S/κ (rk, L, c2) be the moduli stack of S-twisted Higgs sheaves with topological ˚ ss,tw data (rk, L, c2) H (S, Q). Then N is an algebraic stack locally of finite type. The stable locus N s,tw N s,tw P := S/κ (rk, L, c2) is an open substack. One can micmic the construction of [43] to show that the moduli stack N ss exists using quot schemes and the GIT theory. We don’t need to go through that construction based on the following: Proposition 3.25. (generalizing [24, Proposition 2.20]) Based on the Diagram 3.4.1, and under the „ equivalence Higgtw (S) Cohtw(X), the Gieseker (semi)stability of the S-twisted Higgs sheaves (E, φ) KS ÝÑ c with respect to OS(1) is equivalent to the Gieseker (semi)stability of the X-twisted torsion sheaves Eφ with ˚ respect to π OS(1). Proof. The proof is similar to [24, Proposition 2.20]. We choose a generating sheaf Ξ for S. From the „ equivalence Higgtw (S) Cohtw(X), any (F, φ1) (E, φ) is equivalent to the X-twisted sheaves KS ÝÑ c Ă F E on X. We have: Ă φ _ _ ˚ _ χ(S, E Ξ ) = χ(S, π˚E Ξ ) = χ(X, E π Ξ ). b φ b φ b So the Gieseker stability are the same. Note that we use the modified stability on S and X by choosing Ξ = OS OS(r 1). The modified stability is the same as the stability on S by using the ‘¨¨¨‘ ´ geometric χg(E) by basic calculations.

4. TWISTED VAFA-WITTEN INVARIANTS-STABLE CASE Let S S be a µ -gerbe over a smooth projective surface S. The obstruction theory of S-twisted Ñ r Higgs sheaves, and the construction of the virtual fundamental class on the moduli stack of stable S-twisted Higgs sheaves are done in [28] following the construction of Tanaka-Thomas [55]. We only review the basic deﬁnitions of the Vafa-Witten invariants. More details can be found in [55], [24].

4.1. Virtual fundamental class. We fix π : X := Tot(KS) S to be the projection from the total Ñ space of the line bundle KS to S. We follow [24, §3.2], [55, §3] to work on families. Let S B be a family of µ -gerby surfaces S, Ñ r i.e., a smooth projective morphism with the fibre a µ -gerbe surface, and let X B be the total space r Ñ of the a line bundle KS/B. We have a diagram X / B ❄❄ ⑧? ❄❄ ⑧ ❄❄ ⑧⑧ ❄ ⑧⑧ ❄ ⑧ X N s,tw N s,tw such that X = Tot(KS). Recall that we := S/κ(rk, P), the moduli stack of Gieseker stable Higgs pairs on the fibre of S B with fixed rank rk 0 and Hilbert polynomial P. For simplicity, Ñ ą we denote by N tw := N s,tw. 22 YUNFENG JIANG

We pick a (twisted by the G -action) universal sheaf E over N tw X. This universal sheaf may m ˆB not exist due to the G -action, but exists on a finite gerbe over N tw X. We use the same π to m ˆB represent the projection π : X S; π : N tw X N tw S. Ñ ˆB Ñ ˆB Since E is flat over N tw and π is affine,

tw E := π˚E on N S ˆB is flat over N tw. E is also coherent because it can be seen locally on N tw, and is also a S-twisted sheaf. Therefore it defines a classifying map: Π : N tw Mtw Ñ by E π˚E; (E, φ) E, ÞÑ ÞÑ where Mtw is the moduli stack of S-twisted coherent sheaves on the fibre of S B with Hilbert Ñ polynomial P. For simplicity, we use the same E over Mtw S and E = Π˚ E on N tw S. Let ˆ ˆ tw tw tw tw pX : N X N ; pS : N S N ˆB Ñ ˆB Ñ be the projections. Then from [24] and [55] we have an exact triangle:

π˚ [¨,φ] (4.1.1) RH om (E , E ) RH om (E, E) RH om (E, E KS). pX ÝÑ pS ÝÑ pS b Taking the relative Serre dual of the above exact triangle we get RH om (E, E)[2] RH om (E, E K )[2] RH om (E , E )[3]. pS Ñ pS b S/B Ñ pX Proposition 4.1. ([55, Proposition 2.21]) The above exact triangle is the same as (4.1.1), just shifted. Then the exact triangle (4.1.1) ﬁts into the following commutative diagram ([55, Corollary 2.22]): H E E H E E H E E R ompS ( , KS/B)0[ 1] / R ompX ( , )K / R ompS ( , )0 O b ´ O O

H E E H E E H E E R ompS ( , KS/B)[ 1] / R ompX ( , ) / R ompS ( , ) O b ´ O O id tr id tr RpS K [ 1] o / RpS K [ 1] RpS OS o / RpS OS ˚ S/B ´ ˚ S/B ´ ‘ ˚ ˚ H E E where ( )0 denotes the trace-free Homs. The R ompX ( , )K is the co-cone of the middle column ´ N K and it will provide the symmetric obstruction theory of the moduli space L,tw of stable trace free fixed determinant Higgs pairs. In Appendix of [24] the authors generalized the perfect obstruction theory as in [55, §5] of the moduli space of stable Higgs sheaves on the surface S to surface DM stacks. Since the µ -gerbe S S r Ñ is a surface DM stack, and the homological algebra and deformation obstruction theory in [55, §5], [´1,0] H E E [24, Appendix] work for twisted sheaves, we skip the detail. The truncation τ R ompX ( , ) defines a symmetric perfect obstruction theory on the moduli space N tw. We consider the natural Gm-action on the total space KS/B with weight one on the fiber. The tw,Gm obstruction theory is naturally Gm-equivariant. From [14], the Gm-fixed locus N inherits a perfect obstruction theory G [´1,0] H E E ´1 m (4.1.2) τ (R om ( , )[2])t LN tw,Gm pX Ñ by taking the fixed part. Therefore it induces a virtual funda mental cycle

tw G vir tw,G [(N ) m ] H˚(N m ). P COUNTING TWISTED SHEAVES AND S-DUALITY 23

The virtual normal bundle is given _ vir [´1,0] H E E ´1 mov [0,1] H E E mov N := τ (R ompX ( , )[2]t ) = τ (R ompX ( , )[1]) which is the derived dual of the moving part. The virtual localized invariant is given by the following 1 vir . N tw Gm vir e(N ) ż[( ) ] Represent Nvir as a 2-term complex [E E ] of locally free G -equivariant sheaves with non-zero 0 Ñ 1 m weights and define Gm ctop (E0) e(Nvir) := H˚((N tw)Gm , Z) Q[t, t´1], Gm ctop (E1) P b ˚ Gm where t = c1(t) is the generator of H (BGm) = Z[t], and ctop denotes the Gm-equivariant top Chern class lying in H˚((N tw)Gm , Z) Q[t, t´1]. bZ[t] Definition 4.2. Let S S be a µ -gerbe over a smooth projective surface S. Fixing a geometric Hilbert Ñ r polynomial P associated with S. Let N be the moduli space of S-twisted stable Higgs sheaves with Hilbert polynomial P. Then the primitive twisted Vafa-Witten invariants of S is defined as:

tw 1 S Q VW ( ) := vir . N tw Gm vir e(N ) P ż[( ) ] Remark 4.3. For the µ -gerbe S ĄS, we have r Ñ ‚ ‚´1 ‚ ‚ Ext (E , E ) = H (KS) H (OS) Ext (E , E ) , X φ φ ‘ ‘ X φ φ K ‚ E E where ExtX( φ, φ)K is the trace zero part with determinant L Pic(S). Hence the obstruction sheaf in the 2 tw GP vir 0,2 0,1 obstruction theory has a trivial summand H (OS). So [(N ) m ] = 0 is h (S) 0. Ifh (S) 0, then ą ‰ tensoring with flat line bundle makes the obstruction theory invariant. Therefore the integrand is the pullback from N tw/Jac(S), which is a lower dimensional space, hence zero. 4.2. SU(rk)-twisted Vafa-Witten invariants. Instead we work on the moduli stack of S-twisted stable sheaves and Higgs sheaves with fixed determinant and trace zero. Let us now fix (L,0) K,tw P Pic(S) Γ(KS), and let N be the fibre of ˆ L N tw/ Pic(S) Γ(K ). ˆ S Then moduli space N K,tw of stable Higgs sheaves (E, φ) with det(E) = L and trace-free φ L P Hom(E, E K ) admits a symmetric obstruction theory b S 0 ´1 H E E K R ompX ( , )K[1]t LN ,tw ÝÑ L from [24, Proposition A.6]. Definition 4.4. Let S S be a µ -gerbe over a smooth projective surface S. Fixing a geometric Hilbert Ñ r N K,tw polynomial P associated with S. Let L be the moduli space of S-twisted stable Higgs sheaves with Hilbert polynomial P. Then define 1 VWtw(S) := . K vir [(N ,tw)Gm ]vir e(N ) ż L We call them the twisted Vafa-Witten invariants for the gauge group SU(rk)/Zrk. 1 Remark 4.5. In [55, §6], Tanaka-Thomas defined the Vafa-Witten invariants VW(S) := N K Gm vir vir [( L ) ] e(N ) K for smooth projective surface S, here NL is the moduli space of stable Higgs sheaves onş S with fixed data (rk, L, c2), whose structure group is SLrk(κ). This is the Vafa-Witten invariants for the gauge group SU(rk). From [45], for an optimal µr-gerbe S-twisted Higgs sheaf (E, φ), the S-twisted torsion free sheaf E corresponds 24 YUNFENG JIANG to an Azumaya algebra on S, and therefore determines a PGLrk-bundle on S, see [27] for twisted Higgs sheaves. In gauge group level, this corresponds to the gauge group SU(rk)/Zrk.

For the µr-gerbe S, it maybe better to ﬁx the K-group class c K0(S) such that the Hilbert tw 1 P K polynomial of c is P. Then VWc (S) = N ,tw Gm vir vir is Vafa-Witten invariant corresponding [( L ) ] e(N ) to c. The inertia stack IS is r copies of the gerbe S, and the K-group class c K (S) is determined ş P 0 by the orbifold Grothendieck-Riemann-Roch formula (3.2.1), therefore one can ﬁx a data (rk, L, c ) 2 P H˚(S, Q). Thus we have 1 (4.2.1) VWtw (S) := . (rk,L,c ) K vir 2 [(N ,tw)Gm ]vir e(N ) ż L

4.3. The Gm-fixed loci. The group Gm naturally acts on KS by scaling the fiber, therefore it induces N K,tw N K,tw an action on the moduli stack L . We discuss the Gm-fixed loci for the moduli stack L .

4.3.1. Case I-Instanton Branch: For the Higgs pairs (E, φ) such that φ = 0, and is Gm-fixed, the S- G Ms,tw twisted sheaf E must be stable. Therefore the m-fixed locus is exactly the moduli space S/κ := Ms,tw S/κ(rk, L, c2) of S-twisted Gieseker stable sheaves on S with rank rk, fixed determinant L and second Chern class c2. The exact triangle in (4.1.1) splits the obstruction theory ´1 ´1 RH om (E , E ) [1]t RH om (E, E KS) [1] RH om (E, E) [2]t pX K – pS b 0 ‘ pS 0 ´1 where t represents the moving part of the Gm-action. Then the Gm-action induces a perfect obstruction theory ‚ H E E EM := R ompS ( , KS)0[1] LMs,tw . b Ñ S/κ The virtual normal bundle vir ‚ N = RH om (E, E KS) t = EM t[ 1]. pS b 0 b ´ Ms,tw ‚ So the twisted Vafa-Witten invariant contributed from S/κ (we can let EM is quasi-isomorphic to E´1 E0) is: Ñ Gm 0 1 cs (E t) vir = G b [Ms,tw ]vir e(N ) [Ms,tw ]vir c m (E´1 t) ż S/κ ż S/κ r b 0 0 cs(E ) + tcs´1(E ) + = ´1 ´1 ¨ ¨ ¨ [Ms,tw ]vir cr(E ) + tcr´1(E ) + ż S/κ ¨ ¨ ¨ Here we assume r and s are the ranks of E´1 and E0 respectively, and r s is the virtual dimension of Ms,tw 0 ´ S/κ. By the virtual dimension consideration, only t coefficient contributes and we may let t = 1, so 1 c (E0) ‚ ‚ Z (4.3.1) vir = ´1 = cvd(EM) . [Ms,tw ]vir e(N ) [Ms,tw ]vir c‚(E ) vd [Ms,tw ]vir P ż S/κ ż S/κ h i ż S/κ This is the signed virtual Euler number of Ciocan-Fontanine-Kapranov/Fantechi-Göttsche, see [13]. The general case of this invariant is studied in [22]. Hence we have the following result:

Proposition 4.6. If deg KS 0, then any stable S-twisted Gm-fixed Higgs pair (E, φ) has Higgs field φ = 0. ď tw Therefore if we fix c = (rk, L, c2), the twisted invariant VWc (S) is the same as the signed virtual Euler number in (4.3.1). Proof. We use the geometric Gieseker stability to replace the general stability in the proof of [55,

Proposition 7.4]. Given a S-twisted Higgs sheaf (E, φ) which is Gm-fixed, the kernel ker(φ) and Im(φ) are both φ-invariant twisted subsheaves. From stability we have g g g p (m) p (m) p (m) E ă Im(φ) ă EbKS COUNTING TWISTED SHEAVES AND S-DUALITY 25 for m 0 unless Im(φ) = 0 or E KS. The degree deg(KS) 0 implies that deg(KS) 0, therefore g ąą g b ď ď p (m) p (m) for m 0. So either Im(φ) = 0 or the whole E KS. The Higgs pair has fixed EbKS ă E ąą b determinant and trace zero, and it can not be the whole E KS. Therefore φ = 0. b Also we have: Proposition 4.7. If deg K 0, then any semistable G -fixed S-twisted Higgs pair (E, φ) has Higgs field S ă m φ = 0. Proof. This is the same as Proposition 4.6. (2) 4.3.2. Case II-Monopole Branch: The second component M corresponds to the case that in the Gm- fixed S-twisted stable Higgs sheaf (E, φ), the Higgs field φ 0. Let (E, φ) be a G -fixed S-twisted ‰ m stable Higgs pair. Since the corresponding Gm-fixed X-twisted stable sheaves Eφ on X are simple, we use [36, Proposition 4.4], [15] to make this stable sheaf Gm-equivariant. But we work on twisted sheaves. We have the twisted version of Martijn [36, Proposition 4.4]. Lemma 4.8. Let E be a X-twisted stable sheaf on X X which is G -invariant, then it is G -equivariant. Ñ m m Proof. Since a X-twisted stable sheaf E is a coherent sheaf on the DM stack X, Martijn’s result holds for DM stacks.

The cocycle condition in the Gm-equivariant deﬁnition for the S-twisted Higgs sheaf (E, φ) corresponds to a Gm-action ψ : G Aut(E) m Ñ such that (4.3.2) ψ φ ψ´1 = tφ t ˝ ˝ t With respect to the Gm-action on E, the torsion free S-twisted sheaf E splits into a direct sum of eigenvalue subsheaves E = E where E is the weight space such that t has by ti, i.e., ψ = diag(ti). ‘i i i t Conversely, for any twisted pair (E, φ), if the Gm action on E induces a weight one action on the Higgs

field φ, then it is fixed by the original Gm action. The fact that the Gm-action on the canonical line bundle KS has weight 1 makes the Higgs field φ ´ decrease the weights, and it maps the lowest weight torsion subsheaf to zero, hence zero by stability. So each E is torsion free and have rank 0. Thus φ acts blockwise through morphisms i ą φ : E E . i i Ñ i´1 These are flags of torsion-free twisted sheaves on S. We are working on µ -gerbes S S over the surface S, the decomposition of S-twisted sheaves r Ñ E = E depends on the gerbe structure. If the µ -gerbe S S is essentially trivial, then in the ‘i i r Ñ decomposition E = E , we believe that the case that all E have rank one can happen. Then φ ‘i i i i define nesting of ideals and this is the nested Hilbert scheme on S, see [55, §8]. If the µ -gerbe S S is not essentially trivial, for instance, it is optimal, then ind(X) = r, r Ñ which means that the minimal rank of locally free S-twisted sheaf is rank r. Then in this case the decomposition E = E must satisfies that each E has rank at least r. ‘i i i Remark 4.9. It is interesting to study the decomposition for the µr-gerbes over quintic surfaces, and compare with the results in [55, §8]. 4.4. Invariants defined by the Behrend function. In [6], over any DM stack X Behrend constructed an integer value constructible function νX, which is called the Behrend function. If a scheme or a DM ‚ stack X admits a symmetric obstruction theory EX and X is proper then from [6, Theorem 4.18],

1 = χ(X, νX). vir ż[X] This shows that the Donaldson-Thomas invariants for Calabi-Yau threefolds are weighted Euler characteristics. More details of Behrend function and Behrend’s theorem can be found in [6], [23]. 26 YUNFENG JIANG

Let S S be a µ -gerbe over a smooth surface S. On the moduli stack N K,tw (the moduli stack of Ñ r L S-twisted stable Higgs sheaves with data (rk, L, c2)), we have the Behrend function K,tw νN : N Z. L Ñ The weighted Euler characteristic of N tw is deﬁned by:

K,tw ´1 χ(N , νN ) = i χ(ν (i)). L ¨ N PZ iÿ tw The Behrend function is constant on the nontrivial Gm-orbit for the Gm action on N , thus we have the localized Behrend function invariant K,tw K,tw G N N m K χ( L , νN ) = χ(( L ) , νN (N ,tw)Gm ). | L

Definition 4.10. Let S S be a µr-gerbe over a smooth projective surface S. Fixing a geometric Hilbert Ñ N K,tw polynomial P associated with S. Let L be the moduli space of S-twisted stable Higgs sheaves with Hilbert polynomial P. Then define tw N K,tw vw (S) := χ( L , νN ). We call them the small twisted Vafa-Witten invariants for the gauge group SU(rk)/Zrk. Remark 4.11. From calculations in [55, §8], [56], Tanaka-Thomas have confirmed that the right Vafa-Witten 1 invariants VW(S) for a surface S is VW(S) := N K Gm vir vir . They also defined the small Vafa-Witten [( L ) ] e(N ) N K invariants vw(S) := χ( L , νN ). The invariantsş vw = VW for Fano and K3 surfaces. But they are not the same for general type surfaces. We also define both invariants, and later on do the calculations for the small twisted vw invariants.

5. JOYCE-SONG TWISTED STABLE PAIRS AND GENERALIZED TWISTED VAFA-WITTEN INVARIANTS Still fix a µ -gerbe S S over a smooth projective surface S. In this section we include the Joyce- r Ñ Song method to count the semistable S-twisted Higgs sheaves. We then define Vafa-Witten invariants for strictly semistable S-twisted Higgs sheaves. Although this is a parallel theory comparing with Tanaka-Thomas in [56], we include the detail construction since we will use the generalized twisted Vafa-Witten invariants to define the SU(r)/Zr-Vafa-Witten invariants for the Langlands dual group 2 SU(r)/Zr. We also prove the S-duality conjecture for P in rank two. We fix our X = Tot(KS) X Ñ which is a µ -gerbe over X = Tot(K ) with class [X] H2(X, µ ). r S P r 5.1. Background to count semistable objects. Let (E, φ) be a S-twisted Higgs sheaf on a µr-gerbe S S. Recall that from Proposition 3.23, there is a spectral X-twisted sheaf Eφ on X = Tot(KS) with Ñ ˚ respect to the polarization OX(1) = π OS(1). From Diagram 3.4.1 and Proposition 3.25, the Gieseker (semi)stability of the twisted Higgs pair (E, φ) is equivalent to the Gieseker (semi)stability of Eφ. Recall if we fix on the surface S, the following data:

rank(E) = rk; c1(E) = c1; c2(E) = c2. for (E, φ), then from [56, §2.1], let ι : S ã X be the inclusion, we have: Ñ c1(Eφ) = rk [S] ¨ rk(rk +1) c (E ) = ι˚(c + c (S)) $ 2 φ ´ 1 2 1 2 rk(rk +1)(rk +2) 2 &’c (E ) = ι˚(c 2c +(rk +1)c c (S) + c (S) ) 3 φ 1 ´ 2 1 ¨ 1 6 1 in H˚(X, Q). Here’[S] represents the Poincar´edual. Let S S be a µ -gerbe over a smooth projective c % Ñ r surface S. If the µ -gerbe S is trivial, i.e., S [S/µ ] with µ globally trivial action, then one can use r – r r the above data to count (E, φ). COUNTING TWISTED SHEAVES AND S-DUALITY 27

In general for the µ -gerbe S S, one can use the K-group class c K (S) K (X) to measure r Ñ P 0 – 0 the topological data. By the orbifold Grothendieck-Riemann-Roch formula for stacks in (3.2.1), c is ˚ ˚ determined by the Chen-Ruan cohomology class in H (S) H (X). For X = Tot(KS), we use CR – CR the modified Hilbert polynomial PΞ(Eφ(m)) and pΞ(Eφ(m)). We only care about classes (charges) α, β K (X). We follow from [56] to assume that P 0 (5.1.1) pΞ(β(m)) = constant pΞ(α(m)) β = constant α. ¨ ñ ¨ to make the Joyce-Song wall crossing formula easy. Aso for α, β K (X), the Calabi-Yau condition is P 0 satisfied: χ(α, β) = dim Exti(α, β) = 0 ÿi which is skew-symmetric. For the Calabi-Yau threefold DM stack X, and a K-group class α K0(X), there is a moduli Mss,tw P stack X/κ(rk, L, c2) of S-equivalence classes of X-twisted semistable torsion sheaves on X, which Mss,tw is isomorphic to the moduli stack S/κ(rk, L, c2) of S-twisted semistable Higgs sheaves on S. We would use Joyce-Song techniques to count them. 5.2. Hall algebras and the integration map. In this section we review the definition and construction of the motivic Hall algebra of Joyce and Bridgeland in [32], [9]. Then we review the integration map. We briefly review the notion of motivic Hall algebra in [9], more details can be found in [9], [32]. Definition 5.1. The Grothendieck ring of stacks K(St /κ) is defined to be the κ-vector space spanned by isomorphism classes of Artin stacks of finite type over κ with affine stabilizers, modulo the relations:

(1) for every pair of stacks X1 and X2 a relation: [X X ]=[X ]+[X ]; 1 \ 2 1 2 (2) for any geometric bijection f : X X , [X ]=[X ]; 1 Ñ 2 1 2 (3) for any Zariski fibrations p : X Y with the same fibers, [X ]=[X ]. i i Ñ 1 2 1 Let [Aκ ] = L be the Lefschetz motive. If S is a stack of finite type over κ, we define the relative Grothendieck ring of stacks K(St /S) as follows: Definition 5.2. The relative Grothendieck ring of stacks K(St /κ) is defined to be the κ-vector space spanned by isomorphism classes of morphisms f [X S], Ñ with X an Artin stack over S of finite type with affine stabilizers, modulo the following relations:

(1) for every pair of stacks X1 and X2 a relation: f \ f f f [X X 1 2 S]=[X 1 S]+[X 2 S]; 1 \ 2 ÝÑ 1 Ñ 2 Ñ (2) for any diagram: g X / X 1 ❅ 2 ❅❅ ⑦⑦ ❅❅ ⑦⑦ f ❅ ⑦⑦ f 1 ❅ ⑦⑦ 2 S,

f1 f2 where g is a geometric bijection, then [X1 S]=[X2 S]; (3) for any pair of Zariski ﬁbrations Ñ Ñ h h X 1 Y; X 2 Y; 1 Ñ 2 Ñ with the same ﬁbers, and g : Y S, a relation Ñ g˝h g˝h [X 1 S]=[X 2 S]. 1 ÝÑ 2 ÝÑ 28 YUNFENG JIANG

Mtw The motivic Hall algebra in [32] and [9] is deﬁned as follows. Let X/κ be the moduli stack of X-twisted coherent sheaves on X. It is an algebraic stack, locally of ﬁnite type over κ, see [43]. The motivic Hall algebra is the vector space Atw Mtw H( ) = K(St / X/κ) equipped with a non-commutative product given by the rule:

f f2 ˝ [X 1 Mtw ] [X Mtw ]=[Z b h Mtw ], 1 ÝÑ X/κ ‹ 2 ÝÑ X/κ ÝÑ X/κ where h is deﬁned by the following Cartesian square:

h M(2) b Mtw Z / / X/κ

(a1,a2)

f1ˆ f2 X X / Mtw Mtw , 1 ˆ 2 X/κ ˆ X/κ

(2) tw with M the stack of short exact sequences in A , and the maps a1, a2, b send a short exact sequence 0 A B A 0 Ñ 1 ÝÑ ÝÑ 2 Ñ tw to sheaves A1, A2, and B respectively. Then H(A ) is an algebra over K(St /κ). Then on the Hall algebra H(Atw), we are interested in the elements: N ss,tw ã tw tw 1N ss,tw : α Higg (S) Cohc (X), α Ñ KS – ss,tw where Nα is the stack of Gieseker semistable X-twisted Higgs sheaves (E, φ) of class α, and 1 ss,tw Nα is the inclusion into the stack of all twisted Higgs pairs on S. We consider its “logarithm”: ℓ ( 1) (5.2.1) ǫ(α) := ´ 1N ss,tw 1N ss,tw. ℓ α1 ˚¨¨¨˚ αℓ ℓ ℓ ě1,(αi)i=1:αi‰0,@i ÿℓ = = pαi pα, i=1 αi α The key point is that ǫ(α) is virtually indecomposableř in the Hall algebra, see [30, Theorem 8.7]. There is a proof using operators on inertia stacks, see [8]. Then the ǫ(α) is a stack function with algebra stabilizers, ind ǫ(α) SF (Cohtw(X), Q) P al c in Joyce’s notation. From [31, Proposition 3.4], ǫ(α) can be written as

a Z BG i ¨ i ˆ m ÿi where a Q, and BG is the classifying stack. After removing the factor BG on can pullback the i P m m Behrend function to write: (5.2.2) ǫ(α) := a ( f : Z BG Cohtw(X)) . i i i ˆ m Ñ c ÿi Then [31, Eqn 3.22] deﬁnes generalized Donaldson-Thomas invariants

(5.2.3) JStw(X) := a χ(Z , f ˚ν) Q α i i i P ÿi where ν : Mtw Z is the Behrend function. This invariants are deﬁned by applying the integration X/κ Ñ map to the element ǫ(α) in (5.2.1). More details of the integration map can be found in [32], [31] and [9]. For DM stacks, see [25]. COUNTING TWISTED SHEAVES AND S-DUALITY 29

G Mtw As explained in [56, §2], the m-action on X S induces an action on the moduli stack X/κ Ñ tw of twisted sheaves on X. The Gm-action can be extended to the Hall algebra H(A ), and hence ǫ(α) tw carries a Gm-action covering the one on Cohc (X). Also in the decomposition ǫ(α) := a ( f : Z BG Cohtw(X)) , i i i ˆ m Ñ c ÿi the Zi’s admit Gm-action so that they are Gm-equivariant and the proof uses Kretch’s stratification of finite type algebraic stacks with affine geometric stabilizers. Then the Behrend function is also

Gm-equivariant, and tw tw Gm Gm ˚ Q (5.2.4) JSα (X)=(JSα (X)) (X) = aiχ(Zi , fi ν Gm ) |Zi P ÿi 5.3. Application to semistable S-twisted Higgs sheaves. We ﬁrst deﬁne:

Deﬁnition 5.3. Let S S be a µ -gerbe; and X = Tot(KS). For α K (X), we deﬁne the S-twisted Ñ r P 0 U(rk)-Vafa-Witten invariants of S by:

tw tw Gm vwα (S) :=(JSα (X)) (X). Remark 5.4. If the semistability and stability coincides, then Ă tw N s,tw N s,tw Gm vwα (S) = χ( α , νN ) = χ ( α ) , νN N s,tw Gm . |( α ) 1(O ) = ( ) In this case, the invariantsĂ vanish only when h S 0. We need to define the twisted SU rk -Vafa-Witten invariants vwtw. 5.3.1. Joyce-Song twisted stable pairs. We introduce the Joyce-Song pair for X-twisted sheaves. In order to define the Joyce-Song pair, we fix a minimal rank X-twisted locally free sheaf ξ on X such that Ξ = r´1ξbi is a generating sheaf. Here a generating sheaf Ξ on X is p-very ample and contains ‘i=0 every irreducible representations of the cyclic group µ . Recall that p : X X is the map to its coarse r Ñ moduli space, and p˚ : Coh(X) Coh(X) is an exact functor. Ñ Fixing a K-group class α K (X) K (S), for m 0, a Joyce-Song twisted pair consists of the P 0 – 0 ąą following data (1) a compactly supported X-twisted coherent sheaf E with K-group class α K (X), and P 0 0 _ (2) a nonzero section s H (p˚(Ξ E)(m)), i.e., a map s : p˚Ξ p˚E(m). P b Ñ Definition 5.5. A Joyce-Song twisted pair (E, s) is stable if and only if

(1) E is Gieseker semistable with respect to OS(1).

(2) if F E is a proper subsheaf which destabilizes E, then s does not factor through p˚F(m) p˚E(m). Ă Ă ‚ We denote such a pair by I = Ξ( m) E . For our µr-gerbe S S, and the Diagram 3.4.1, t ´ Ñ u Ñ ě1 X X is also a µ -gerbe. For any X-twisted coherent sheaf E on X, we have H (p˚E(m)) = 0 for Ñ r large m 0. We then take P tw := P tw (X) to be the moduli stack of X-twisted stable Joyce-Song ąą (rk,L,c2) pairs I‚ = Ξ( m) E t ´ Ñ u on X. This moduli stack can be constructed as the moduli stack of simple complexes over the gerbe X, see [3], [31]. The moduli stack P tw is not compact, but admits a symmetric obstruction theory since we can take it as the moduli space of simple complexes on a Calabi-Yau threefold DM stack X. The invariant we tw tw use is Behrend’s weighted Euler characteristic P (m) := χ(P , νP ). Under the Gm-action: (rk,L,c2)

tw tw,Gm Gm P (m) = P r (m) = χ P , νP PGm . (rk,L,c2) (rk,L,c2) | r r 30 YUNFENG JIANG

For generic polarization, Joyce-Song invariants JStw(X) Q satisfy the following identities [31, α P Theorem 5.27]: ℓ ℓ ( 1) tw χ(αi(m)) tw (5.3.1) P (m) = ´ ( 1) χ(αi(m)) JS (X). (rk,L,c2) ℓ! ´ ¨ ¨ αi ℓ ℓ i=1 ě1,(αi=δiα)i=1: ÿℓ ź r δią0, i=1 δi=1 Since P tw (m) is deformation invariant,ř which implies that JStw is also deformation invariant. For (rk,L,c2) α general OS(1), the wall crossing formula is complicated as in [31, Theorem 5.27]. If semistability coincidesr with stability for twisted sheaves E, the moduli stack P tw is a Pχ(α(m))´1-bundle over (rk,L,c2) the moduli stack N tw of X-twisted torsion sheaves E. The Behrend function νP of P tw (X) is (rk,L,c2) (rk,L,c2) the pullback of the Behrend function on N tw multiplied by ( 1)χ(α(m))´1. Therefore: (rk,L,c2) ´ P tw χ(α(m))´1 (5.3.2) (m)=( 1) χ(α(m)) vw( )(S) (rk,L,c2) ´ ¨ ¨ rk,L,c2 which is the first term ℓ = 1 in (5.3.1). In general the formula expresses P tw (m) in terms of r Ă (rk,L,c2) rational corrections from semistable X-twisted sheaves E on X. r Remark 5.6. Before we go to generalized twisted SU(rk)-Vafa-Witten invariants, it is worth mentioning that the wall crossing formula (5.3.1) above for P tw (m) works for X-twisted sheaves in [31, Theorem 5.27]. (rk,L,c2) B E The reason is that Joyce-Song prove it for categories pα , whose objects are X-twisted semistable sheaves with reduced Hilbert polynomial pα, and a vectorr space V together with a linear map V H0(E(m)) Ñ with the same Euler forms χ. Joyce-Song’s techniques [31, Theorem 5.27] work for abelian categories with stability conditions, thus the wall crossing techniques work for this twisted category. 0,1 5.3.2. Generalized twisted SU(rk)-Vafa-Witten vw-invariants. For the µr-gerbe S S, if h (S) 0, tw Ñ ą then vw defined before vanish because of the action of Jac(S). We follow from [56] to fix the determinant det(E). Let us fix a line bundle L Pic(S), and use the map: Ă P det ˝π˚ Cohtw(X) Pic(S). c ÝÑ Let us define tw L ´1 Coh (X) :=(det π˚) (L). c ˝ We can restrict Joyce’s stack function to this restricted category. For any stack function F := ( f : U Cohtw(X)) we define: Ñ c L tw L tw F = f : U tw Coh (X) Coh (X) ˆCohc (X) c Ñ c which is 1 tw L F, where is the ordinary product [31, Definition 2.7]. Then ǫ(α) in (5.2.2) becomes: Cohc (X) ¨ ¨ L tw L tw (5.3.3) ǫ(α) := a f : Z tw Coh (X) /Gm Coh (X) . i i i ˆCohc (X) c Ñ c ÿi Then applying the integration map again and we get the fixed determinant generalized Donaldson- Thomas invariants: L,tw tw L ˚ JS (X) := a χ Z tw Coh (X) , f ν . α i i ˆCohc (X) c i ÿi We also compute it using localization:

L,tw L,tw Gm Gm tw L ˚ (5.3.4) JS (X)=(JS ) (X) = a χ Z tw Coh (X) , f ν . α α i i ˆCohc (X) c i ÿi COUNTING TWISTED SHEAVES AND S-DUALITY 31

Definition 5.7. We define the twisted SU(rk) generalized Vafa-Witten invariants as: 0 vwtw (S) :=( 1)h (KS) JSL,tw (X) Q. (rk,L,c2) ´ (rk,L,c2) P As explained in [56, §4], we did not restrict to the trace zero tr φ = 0, but in the SU(rk)-moduli 0 stack case we did. The difference is only a Behrend function sign ( 1)dim H (KS). If h0,1(S) = 0 and ´ OS(1) is generic, then the wall crossing formula (5.3.1) becomes: ℓ ℓ ( 1) 0 (5.3.5) P tw(m) = ´ ( 1)χ(αi(m))+h (KS) χ(α (m)) vwtw(S). α ℓ! ´ ¨ i ¨ αi ℓ ℓ i=1 ě1,(αi=δiα)i=1: ÿℓ ź r δią0, i=1 δi=1 If h0,1 0, and O (1) generic,ř we should use Joyce-Song pairs (E, s) with fixed determinant ą S det(π˚E) L Pic(S). – P B B Ψ pα Joyce-Song use the category pα before, and we apply the integration map to: L tw L ǫ := ǫα tw Coh (X) α,1 ,1 ˆCohc (X) c r where ǫα,1 is the stack function (13.25), (13.26) in [31, Chapter 13]. Since ǫα,1 is virtually indecomposable, we get P tw (m) := χ(P tw, νP ), α=(rk,L,c2) P tw P tw E E where := α is the moduli stack of stable Joyce-Song twisted pairs ( , s) with det(π˚ ) = L. Hence P tw PGm P tw Gm (5.3.6) α (m) = α (m) = χ(( )α , νP PGm ). | α For generic O (1), we get ([56, Proposition 4.4]): if h0,1(S) 0, the invariants (P tw)Gm (m) determines S ą α the twisted SU(rk)-Vafa-Witten invariants vwtw of Definition 5.7 by: 0 (5.3.7) P tw(m)=( 1)h (KS)( 1)χ(α(m))´1χ(α(m)) vwtw(S). α ´ ´ α Remark 5.8. The proof of (5.3.7) is the same as [56, Proposition 4.4], and the basic reason is that if h0,1(S) ą 0, then the Jacobian Jac(S) is an abelian variety which acts on the moduli stacks to force the Euler characteristic to be zero. 5.4. Generalized twisted SU(rk)-Vafa-Witten invariants VWtw. Fix a µ -gerbe S S, and the µ - r Ñ r gerbe X X as in the diagram (3.4.1). In this section we generalize the arguments in [56, §6] to Ñ tw twisted generalized SU(rk)-Vafa-Witten invariants VW . For m 0, fixing α =(rk, L, c ), L Pic(S), the Joyce-Song twisted pair (E, s) is defined in §5.3.1, ąą 2 P where E is a X-twisted sheaf on X corresponding to a S-twisted Higgs sheaf (E, φ) on S. Let PK,tw P tw α Ă α E P tw be the moduli stack of twisted stable pairs with det(E) det(π˚ ) = L, tr φ = 0. On α , there is a symmetric obstruction theory, see [31, Chapter 12]. The tangent-obstruction complex is given by: ‚ ‚ ‚ s RH omX(I , I )0[1] at the point I = OX E . By [55, §5], or [24, Appendix] we can modify it to Kt,tw ÝÑ u a symmetric obstruction theory on Pα . Actually we have: ‚ ‚ ‚ ‚ ˚ ě1 ď1 RH omX(I , I ) = RH omX(I , I ) H (OX) H (OS) H (KS)[ 1], K ‘ ‘ ‘ ´ ‚ where we actually removed the deformation-obstruction theory of det(I ) OX, of det(π˚E) OS – – and of tr φ Γ(KS). Since R.Thomas mentioned that this can be done in families, we omit the details. P We apply the Gm-virtual localization [14] to define: Definition 5.9. 1 PK,tw(m) := . α K vir P ,tw Gm vir e(N ) ż[( α ) ] 32 YUNFENG JIANG

We micmic [56] to conjecture: Conjecture 5.10. 0,1( ) = 0,2( ) = tw(S) If H S H S 0, there exist rational numbers VWαi such that ℓ ℓ ( 1) PK,tw(m) = ´ ( 1)χ(αi(m)) χ(α (m)) VWtw(S). α ℓ! ´ ¨ i ¨ αi ℓ ℓ i=1 ě1,(αi=δiα)i=1: ÿℓ ź δią0, i=1 δi=1 for m 0. When either of H0,1(S)řor H0,2(S) is nonzero, we only take the ﬁrst term in the sum: ąą PK,tw(m)=( 1)χ(α(m))´1 χ(α(m)) VWtw(S). α ´ ¨ ¨ α This conjecture is similar to Conjecture 6.5 in [56]. We hope that Laarakker’s method [38] can prove this conjecture for gerbes. The reason to make this conjecture can be found in [56, §6]. We expect that the conjecture makes sense for µ -gerbes S S, since the deformation and obstruction theory for r Ñ sheaves on gerbes behave like sheaves on surfaces although the summand in the conjecture should depend on the µr-gerbe structure on S. In [29], we prove this conjecture for µr-gerbes on K3 surfaces tw tw and also the equality VWα (S) = vwα (S) for a K3 gerbe. Finally in this section we provide two results for the µ -gerbe S S generalizing [56, Proposition r Ñ 6.8, 6.17], which will be useful later for the calculations.

K,tw Proposition 5.11. If the semistable X-twisted sheaves in Nα coincide with stable X-twisted sheaves, then tw 1 Conjecture 5.10 is true and VWα Q deﬁned by N K,tw vir vir . P [ α ] e(N ) Proof. The proof is the same as [56, Propositionş 6.8], since the deformation of Joyce-Song twisted pairs I‚ and E are similar to the general pairs and sheaves on the DM stack X.

Proposition 5.12. Suppose that K 0, then Conjecture 5.10 is true and VWtw(S) = vwtw(S). S ă α α Proof. The proof is also the same as [56, Proposition 6.17].

2 5.5. The SU(r)/Zr-Vafa-Witten invariants and proof in rank two for P . In this section we define tw the SU(r)/Zr-Vafa-Witten invariants using the twisted SU(rk)-Vafa-Witten invariants VW . We fix a smooth projective surface S. Let r Zą0 be an integer. The µr-gerbes S S are classified 2 2 P Ñ by H (S, µr). For any g H (S, µr), let Sg S be the µr-gerbe corresponding to this g. Since the P 2 Ñ étale cohomology group H (S, µr) is finite, we have finite number of equivalent µr-gerbes over S. Now fixing a K-group data K (S ) given by α =(r, L, c ) H˚(S , Q), we have the moduli stack 0 g 2 P g N K,tw L (Sg) of Sg-twisted semistable Higgs sheaves with data α; and we have the generalized twisted tw Vafa-Witten invariants VWα defined in Definition 4.4 in the stable case, and Conjecture 5.10 in the tw semistable case. We use all the semistable invariants VWα . Definition 5.13. Fix an r Z , for any µ -gerbe p : S S corresponding to g H2(S, µ ), let P ą0 r g Ñ P r L Pic(S ) let P g tw c Z L(Sg, q) := VW (Sg)q 2 r, (r,L,c2) c ÿ2 be the generating function of the twisted Vafa-Witten invariants. Let us fix a line bundle L Pic(S), and define for any essentially trivial µ -gerbe S S corresponding to P r g Ñ the line bundle L Pic(S), L := p˚L L ; for all the other µ -gerbe S S, keep the same L = p˚L. g P g b g r g Ñ g For L Pic(S), let L H2(S, µ ) be the image under the morphism H1(S, O˚) H2(S, µ ). We define P P r S Ñ r 2πig¨L r Zr,L(S,SU(r)/Zr; q) := e Zr,Lg (Sg, q). 2 gPHÿ(S,µr) We call it the partition function of SU(r)/Zr-Vafa-Witten invariants. COUNTING TWISTED SHEAVES AND S-DUALITY 33

Conjecture 5.14. For a smooth projective surface S, the partition function of SU(r)-Vafa-Witten invariants

( ( ) ) = ( ) c2 Zr,L S,SU r ; q VW(r,L,c2) S q c ÿ2 and the partition function of SU(r)/Zr-Vafa-Witten invariants Zr,L(S,SU(r)/Zr; q) satisfy the S-duality conjecture in Conjecture (1.1.2).

We include a proof of this conjecture for P2 in rank two.

2 2 5.5.1. The partition function for P . We consider the projective plane P . Let MP2 (2, c1, c2) be the moduli space of stable torsion free sheaves of rank 2, ﬁrst Chern class c1 and second Chern class c . Since K 2 0, any semistable Higgs sheaf (E, φ) will have φ = 0. Therefore the moduli space 2 P ă of stable Higgs sheaves NP2 (2, c1, c2) is isomorphic to MP2 (2, c1, c2). Also the space MP2 (2, c1, c2) is smooth and the Vafa-Witten invariant is just (up to a sign) the Euler characteristic of the moduli space. Then we introduce 2 P c2 Z (q) = χ(M 2 ( c c ))q (5.5.1) c1 P 2, 1, 2 c ÿ2 p Let NP2 (2, c1, c2) be the moduli space of stable vector bundles of rank 2, ﬁrst Chern class c1 and second Chern class χ. Let 2 vb,P c2 ( ) = ( 2 ( )) Zc1 q e NP 2, c1, c2 q c ÿ2 be the partition function. Then from [63], [36],

1 P2 q 8 vb,P2 Zc (q) = Zc (q) 1 η(q)χ(P2) ¨ 1 where η(q) is the Dedekind eta function.p To state the result we introduce some notations. First let H(∆) be the Hurwitz class numbers, i.e., H(∆) is the number of positive deﬁnite integer binary quadratic forms AX2 + BXY + CY2 such that B2 4AC = ∆ and weighted by the size of its automorphisms group. Let σ (n) be the divisor ´ ´ 0 function.

Theorem 5.15. ([34], [63], [36]) We have:

1 2 3 8 1 4 c1+ 2 c1+2 4 ń vb,P2 q n=1 3H(4n 1)q ; (c1 odd); Z (q) = 1 2 3 ¨ ´ c1 c + c1+2 8 1 ń $q 4 1 2 3 H(4n) σ0(n) q ; (c1 even). & ¨ řn=1 ´ 2 ř % vb,P2 In the case c1 is odd, by the work of D. Zagier [66] we see that Zc1 (q) is the holomorphic part Γ ´1 of a modular form of weight 3/2 for 0(4) (up to replacing q by q and up to an overall power of q in front). In the case c1 is even one only obtains modularity after correctly adding strictly semistable sheaves to the moduli space. Their contribution turns out to cancel the sum of divisors term. In order to compare with other partition functions later we introduce: 8 vb,P2 1 c2+ 3 c +2 1 ń (5.5.2) Z (q) := q 4 1 2 1 3H(4n 1)q 4 ; c1,odd ¨ ´ = nÿ1 and 8 vb,P2 1 c2+ 3 c +2 1 ń (5.5.3) Z (q) := q 4 1 2 1 3 H(4n) σ (n) q . c1,even ¨ ´ 2 0 = nÿ1 34 YUNFENG JIANG

5.5.2. The partition function for P(2,2,2). In [15], the authors generalize the calculation of the moduli of stable torsion free sheaves on smooth toric variety to weighted projective spaces P(a, b, c), which is a special toric DM stack. The calculation uses and generalizes the toric method in [36] to this toric DM stack. We omit the detail calculation and only include the calculation results for P(2,2,2). P P a b c In general the weighted projective plane (a, b, c) is a µd-gerbe ( d , d , d ) where d = gcd(a, b, c). 2 In the case of P(2,2,2) which is a µ2-gerbe over P , the partition function will depend on the choice of the component of the inertia stack IP(2,2,2) = P(2,2,2) P(2,2,2). We use λ = 0, or 1 to distinct Y these two components. Now we list the results for P(2,2,2). In this case the first Chern class c is always even. Let λ 1 P 0,1 index the component in the inertia stack IP(2,2,2) = P(2,2,2) P(2,2,2). Let N (2, c , χ) t u Y P(2,2,2) 1 be the moduli space of stable vector bundles of rank 2, first Chern class c1 and second Chern class c2. Let P vb, (2,2,2) c2 Z (q) = χ(NP (2, c , c ))q c1,λ (2,2,2) 1 2 c ÿ2 be the partition function. Theorem 5.16. ([15, Theorem 1.2]) Since c is even, there are two cases c 0(mod 4) or c 2(mod 4). 1 1 ” 1 ” We have P( ) Zvb, 2,2,2 (q) = c1,0 P2 1 2 3 8 vb, 16 c1+ 4 c1+2 1 ń Z c1 (q) = q n=1 3(H(4n) 2 σ0(n))q ; (c1 0 mod 4); 2 ¨ ´ ” vb,P2 1 2 3 8 1 $ 16 c1+ 4 c1+2 4 ń Z c1 (q) = q řn=1 3(H(4n 1)q ; (c1 2 mod 4) & 2 ¨ ´ ” and % ř P( ) Zvb, 2,2,2 (q) = c1,1 vb,P2 1 c1 2 3 c1 8 1 4 ( 2 +1) + 2 ( 2 +1)+2 4 ń Z c1 (q) = q n=1 3(H(4n 1)q ; (c1 0 mod 4); 2 +1 ¨ ´ ” vb,P2 1 c1 2 3 c1 8 $ 4 ( 2 +1) + 2 ( 2 +1)+2 1 ń Z c1 (q) = q řn=1 3(H(4n) 2 σ0(n))q ; (c1 2 mod 4). & 2 +1 ¨ ´ ” ř Remark 5.17.% In the case c1 is even, from [15] one only obtains modularity after correctly adding strictly semistable sheaves to the moduli space. Their contribution turns out to cancel the sum of divisors term. Thus in the following when checking the S-duality, we can ignore the divisor functions. 5.5.3. S-duality. From (4.30) of [61, §4], by a result of Zagier [66], let

1 2 n ´ 2 2 ´n f0 = 3H(4n)q + 6τ2 β(4πn τ2)q ně0 PZ ÿ nÿ and 1 n´ 1 ´ 1 2 ´(n+ 1 )2 f = 3H(4n 1)q 4 + 6τ 2 β(4π(n + ) τ )q 2 1 ´ 2 2 2 ną0 PZ ÿ nÿ 2πiτ where q , and τ2 = Im(τ), and 8 1 ´ 3 ´ut β(t) = u 2 e du. 16π ż1 vb,P2 From [66], these functions are modular, but not holomorphic. Hence Zc1 (c1 even or odd) is the homomorphic part of the non-holomorphic modular functions above. Also by Zagier, see [61, Formula (4.31)], under τ 1 , we have: ÞÑ ´ τ 1 f ( ) τ 3 1 1 1 f (τ) 0 τ 2 0 (5.5.4) ´ 1 =( ) . f1( ) i ¨ ´?2 1 1 f1(τ) ´ τ ´ COUNTING TWISTED SHEAVES AND S-DUALITY 35

This is the transformation conjecture (1.1.1). We know that f0 is invariant under T and f1 is invariant 4 4 under T . Therefore f0 is invariant under ST S. To check the S-duality, we choose the case c1 = 0 or 2, from Theorem 5.16 and Theorem 5.15 we calculate: P( ) 2 Zvb, 2,2,2 (q) = Zvb,P (q) = q2 8 3(H(4n) 1 σ (n))qń; 0,0 0 n=1 ´ 2 0 vb,P(2,2,2) vb,P2 15 8 1 ń $Z (q) = Z (q) = q 4 3(H(4n 1)q 4 ; 2,0 1 ř¨ n=1 ´ ’ vb,P(2,2,2) vb,P2 15 8 1 ń ’Z (q) = Z (q) = q 4 3(H(4n 1)q 4 ; &’ 0,1 1 ¨ řn=1 ´ vb,P(2,2,2) vb,P2 6 8 1 ń Z2,1 (q) = Z2 (q) = q nř=1 3(H(4n) 2 σ0(n))q . ’ ´ Then we check’ that under transformation τ 1 , we have % ÞÑ ´řτ ´2 vb,P2 ´2 vb,P(2,2,2) ´ 15 vb,P(2,2,2) (5.5.5) q Z (q) q Z (q) + q 4 Z (q). ¨ 0 ÞÑ 0,0 ¨ 0,1 and ´ 15 vb,P2 ´ vb,P(2,2,2) ´ 15 vb,P(2,2,2) (5.5.6) q 4 Z (q) q 6Z (q) q 4 Z (q). ¨ 1 ÞÑ 2,1 ´ ¨ 2,0 Definition 5.18. We define

P2 1 ´2 vb,P(2,2,2) ´ 15 vb,P(2,2,2) Z (SU(2)/Z ; τ) := q Z (q) + q 4 Z (q) 0 2 2 ¨ ¨ 0,0 ¨ 0,1 and P2 1 ´6 vb,P(2,2,2) ´ 15 vb,P(2,2,2) Z (SU(2)/Z ; τ) := q Z (q) q 4 Z (q) 1 2 2 ¨ 2,1 ´ ¨ 2,0 Then from the above calculations in (5.5.5), we have Theorem 5.19. We define 2 2 ZP (SU(2); τ) = q´2 Zvb,P (q) 0 ¨ 0 and P2 ´ 15 vb,P2 Z (SU(2); τ) = q 4 Z (q) 1 ¨ 1 Under the S-transformation τ 1 , we have: ÞÑ ´ τ 3 P2 1 ´ 3 τ 2 P2 Z SU(2); = 2 2 Z (SU(2)/Z ; τ) 0 ´ τ ˘ i 0 2 and 3 P2 1 ´ 3 τ 2 P2 Z SU(2); = 2 2 Z (SU(2)/Z ; τ) 1 ´ τ ˘ i 1 2 And the S-duality conjecture (1.1.1) holds. Proof. The group is H2(P2, µ ) µ , where 0 µ corresponds to the trivial µ -gerbe P2/µ = 2 – 2 P 2 2 2 P2 Bµ , and 1 µ corresponds to the nontrivial µ -gerbe P(2,2,2) over P2 given by the line ˆ 2 P 2 2 bundle OP2 ( 1). ´ vb,P(2,2,2) We prove the first transformation formula. The partition function Z0,0 (q) is the same as the 2 vb,[P /µ2] 2 partition function Z0,0 (q) of the trivial µ2-gerbe [P /µ2]. This is because the partition function 2 vb,[P /µ2] Z0,0 (q) corresponds to counting torsion free rank two sheaves with trivial µ2-action. The vb,P(2,2,2) vb,P(2,2,2) partition function Z0,1 (q) is the same as the partition function Z2,0 (q) from Theorem 5.16 and the above calculation. We explain this a bit. The inertia stack IP(2,2,2) = P(2,2,2) P(2,2,2) , 0 Y 1 where the first component corresponds to 0 µ , and the second component corresponds to 1 µ . P 2 P 2 Our moduli of twisted sheaves actually counts sheaves on the first component. The moduli space of µ -gerbe P(2,2,2)-twisted sheaves with data (2, p˚O( 1), c ) is isomorphic to the moduli space of 2 ´ 2 twisted sheaves with data (2,0, c2) on the second component. The second formula is similar. Thus from the above S-transformation in (5.5.4) and (5.5.5), the theorem follows. 36 YUNFENG JIANG

6. S-DUALITY CONJECTURE FOR K3 SURFACES In this section we study Conjecture 5.14 for smooth K3 surfaces. We prove the S-duality conjecture of Vafa-Witten in all the prime ranks r. 6.1. K3 surfaces- Picard numbers, Mukai Lattice and Brauer group. This section serves some basic knowledges on smooth K3 surfaces. Our reference is [20]. Let S be a smooth projective K3 surface. The cohomology H˚(S, Z) is zero on odd degrees, and H0(S, Z) = H4(S, Z) Z, H2(S, Z) = Z22. p,q q Ωp – The Hodge diamond for the Hodge numbers h = dim H (S, S) is given by: h0,0 1 h1,0 h0,1 0 0 h2,0 h1,1 h0,2 = 1 20 1 h2,1 h1,2 0 0 h2,2 1 Let Pic(S) be the Picard group of S. The Néron-Severi group of S is the quotient NS(S) = Pic(S)/ Pic0(S), where Pic0(S) is the subgroup of all line bundles that are algebraically equivalent to zero. The group Num(S) is defined as the quotient of the Picard group Pic(S) by the kernel of the intersection form on H˚(S, Z). We say that line bundles from the kernel are numerically trivial. The rank of NS(S) is called the Picard number and is denoted by ρ(S). For algebraic K3 surface S. Pic(S) NS(S) Num(S). – – The Picard number ρ(S) satisfies 0 ρ(S) 20 ď ď for a smooth complex K3 surface S. Over character zero field κ, a smooth K3 surface S with Picard number ρ = 20 is called a singular K3. Over the field κ with positive characteristic p 0, a K3 surface ą always has Picard rank ρ(S) 1, and a K3 surface with Picard number ρ(S) = 22 = h1,1 is called a ě supersingular K3 surface, see [47]. On the cohomology H˚(S, Z), we review the Mukai lattice as in [64]. Definition 6.1. Define a symmetric bilinear form on H˚(S, Z):

x, y = x_y x y ´ żS where x_ is the dual of the class x. We call this lattice the Mukai lattice. Remark 6.2. If x = x + x + x , for x H2i(S, Z), y = y + y + y , for y H2i(S, Z), then : 0 1 2 i P 0 1 2 i P _ H˚(S, Z) H˚(S, Z) is the homomorphism sending x to x x + x . Then Ñ 0 ´ 1 2 x, y = x y x y x y . x y 1 1 ´ 0 2 ´ 2 0 Let E be a coherent sheaf on S, we deﬁne

v(E) := Ch(E)?tdS = Ch(E)(1 + ω) where ω is the fundamental class of S and Ch(E) is the Chern character of E. Then Riemann-Roch theorem says that χ(E, F) = v(E), v(F) , ´x y where E and F are coherent sheaves on S. This Mukai lattice has a decomposition: H˚(S, Z) = H2(S, Z) H0(S, Z) H4(S, Z)=( E )‘2 H4, ‘ ‘ ´ 8 ‘ where H is the hyperbolic lattice. Some properties of the Mukai lattices like isometries can be found in [64]. We talk about the Brauer group of S. According to [19], also from de Jong, 1 2 O˚ Br(S) = Br (S) = H (S, S )tor. COUNTING TWISTED SHEAVES AND S-DUALITY 37

Consider the exact sequence (¨)r 1 µ O˚ O˚ 1 Ñ r ÝÑ S ÝÑ S Ñ and taking long cohomology we get an exact sequence:

1 ˚ 1 ˚ 2 2 ˚ 1 H (S, O )/r H (S, O ) H (S, µ ) H (S, O ) ´ 1. Ñ S ¨ S ÝÑ r ÝÑ S r tor Ñ This is true for any r Z . Therefore Br(S)=(Q/Z)22´ρ(S), where ρ(S) is the Picard number. P ě0 Finally we have H2(S, µ ) (Z )22. r – r Let us pay attention to the case r = p for a prime number p. Then in this case,

H2(S, µ ) = (Z )22 = p22. | p | | p | Proposition 6.3. The number of equivalent classes of non-essentially trivial µ -gerbes on S is p22 pρ(S). p ´ ρ(S) Proof. The essentially trivial µp-gerbes on S is given by the Picard number ρ(S), which is p . Thus the result just follows.

Remark 6.4. In the case p = 2, the number of equivalent optimal µ -gerbes on S is exactly 222 2ρ(S). 2 ´ 6.2. Tanaka-Thomas’s Vafa-Witten invariants for K3 surfaces. Let S be a smooth K3 surface. Then fixing a class α = (r, L, c ) H˚(S, Z), the Vafa-Witten invariants (stable ones or the generalized 2 P ones) VWα(S), vwα(S) are defined in [55], [56]. They have the same definitions as in Definition 4.4, Definition 4.10, Definition 5.7, and Conjecture 5.10 when taking no gerbe structures on S and omit the twist. Also there is a similar conjecture for VWα(S) as in Conjecture 1.2 in [56]. This conjecture is true for K3 surfaces [48]. Thus

VWα(S) = vwα(S). We recall some results of [56, §5].

Proposition 6.5. ([56, Proposition 5.1], generalizing the stable Higgs pair case) If (E, φ) is a Gm-ﬁxed Higgs pair by the action of scaling the Higgs ﬁeld φ, then E admits an algebraic Gm-action

Ψ : G Aut(E) m Ñ such that Ψ φ Ψ´1 = tφ for all t G . t ˝ ˝ P m

Proposition 6.6. ([56, Lemma 5.6]) Let (E, φ) be a Gm-ﬁxed Gieseker semistable Higgs pair, then E is Gieseker semistable. Moreover, either φ = 0 or c2(E) = 2k is even and, up to twist by some power of t, we have 0 0 (6.2.1) E = I I t´1, φ = Z ‘ Z ¨ 1 0 for some length k subscheme Z S. Ă

Proposition 6.7. ([56, Proposition 5.9], [48]) Let (E, φ) be a Gm-ﬁxed Gieseker semistable Higgs pair on a smooth K3 surface S with ﬁxed determinant L and tr φ = 0, then the Behrend function at this point

νN = 1. ´ 6.3. Twisted Vafa-Witten invariants-essentially trivial gerbes. In this section we study the twisted Vafa-Witten invariants for essentially trivial µ -gerbes S S over the K3 surface S. r Ñ 38 YUNFENG JIANG

6.3.1. Trivial µ -gerbes on S. Recall that a µ -gerbe S S is called essentially trivial if the class [S] r r Ñ is in the image of the map: H1(S, O˚) H2(S, µ ). Thus from §2.2, the gerbe S is given by a line S Ñ r bundle L Pic(S) on S. If L is non-trivial, we call S a nontrivial essentially trivial µ -gerbe, otherwise, P r if L is a trivial line bundle on S, the gerbe S =[S/µr] is trivial, where µr acts globally trivial on S. For such a trivial µ -gerbe S S, any S-twisted sheaf E on S is actually a sheaf and is pullback from S. r Ñ We have:

Proposition 6.8. Let S = [S/µr] S is a trivial µr-gerbe over a K3 surface S. Then the moduli stack N ss,tw O Ñ N ss O S/κ(r, , c2) of S-twisted semistable Higgs sheaves is isomorphic to the moduli stack S/κ(r, , c2) of Gieseker semistable Higgs sheaves on S. Proof. This is from the fact that any S-twisted semistable sheaf or Higgs sheaf on S is a sheaf on S, which is a pullback from a semistable sheaf on S. So the moduli spaces are isomorphic.

tw tw Then the twisted Vafa-Witten invariants VW (S) = vw (S) for trivial µr-gerbe S are the same as the Vafa-Witten invariants VW(S) and vw(S) as in [56]. 6.3.2. Non-trivial essentially trivial µ -gerbes. Let S S be a non-trivial essentially trivial µ -gerbe r Ñ r over a K3 surface S. First we make Proposition 3.15 rigorous in the K3 surface case.

Proposition 6.9. Suppose that p : S S is a non-trivial essentially trivial µr-gerbe corresponding to the Ñ tw line bundle L Pic(S), then the moduli stack N ss, (r, L, c ) of S-twisted Higgs sheaves on S is isomorphic P S/κ 2 N ss,tw O to the moduli stack S/κ (r, , c2) of S-twisted Higgs sheaves on S. 1 ˆ Proof. Let S = [L] r = [Tot(L )/Gm], be the r-th root stack for the line bundle L, which is the gerbe S we want. Let L be the universal r-th root of the line bundle L. Then the functor E E L_ ÞÑ b yields an equivalence of categories from S-twisted sheaves on S to sheaves on S. Hence also yields an equivalence of categories from S-twisted Higgs sheaves on S to Higgs sheaves on S. The semistability of a S-twisted Higgs sheaf (E, φ) translates into L(r γ)-twisted semistability of E L_. 1 _ ¨ b Here the line bundle γ Pic(S) r Z is the line bundle L on S. P Mss,twb L Mss,γ L Thus the moduli stack S/κ(r, , c2) is isomorphic to the stack S/κ(r, (rγ), c2) of γ-twisted semistable sheaves on S of rank r, determinant L(rγ) and second Chern class c2. This is also true for the moduli stack of Higgs sheaves. Now we restrict ourself to the rank 2 case. Let S S be a non-trivial essentially trivial µ -gerbe. Ñ 2 Proposition 6.10. Let (E, φ) be a S-twisted semistable rank 2 Higgs sheaf with class (2, L, c2). Then the torsion free sheaf E can not split into sum of line bundles. Proof. First since our torsion free sheaf E has determinant det(E) = L, where L is the deﬁning line bundle for the µ -gerbe S. Our gerbe [S] H2(S, µ ) is a nontrivial element. Thus r P 2 c (E) = c (det(E)) [S] mod 2. 1 1 ” One can think of this as the second Stiefel-Whitney class of E on S. Suppose that E can be split into E = E1 E2 such that each Ei is of rank one. Also det(E) = L 1 1 ‘ 1 implies that E = E E = L 2 L 2 , in which L 2 is also non-trivial. Then c (E) = c (E ) + c (E ) 1 ‘ 2 ‘ 1 1 1 1 2 and after modulo mod 2 we get that c (E) 0 mod 2, which is a contradiction. 1 ” Remark 6.11. Another way to see this is that E, as a S-twisted torsion-free sheaf or vector bundle, can be taken as an SO(3) = SU(2)/Z2-bundle. Then an SO(3)-bundle with non-trivial Stiefel-Whitney class w2(E) = c1(E) mod 2 can not split. Thanks to Yang Li for talking about this point. Now we are ready to calculate the twisted Vafa-Witten invariants. COUNTING TWISTED SHEAVES AND S-DUALITY 39

Proposition 6.12. Let S S be a non-trivial essentially trivial µ -gerbe over the K3 surface S corresponding Ñ 2 to the line bundle L Pic(S). Then we have P tw tw vw2,L,0(S) = 0; vw2,L,1(S) = 0. For k 1 we have ě tw 4k´3 vw2,L,2k(S) = χ(Hilb (S)); $ ’ tw 4k´1 &vw2,L,2k+1(S) = χ(Hilb (S)), n where χ(Hilb (S)) is the topological%’ Euler number of the Hilbert scheme of n points on S. Proof. From Deﬁnition 5.7,

0 tw h (KS) L,tw vw L (S) :=( 1) JS L (X). (2, ,c2) ´ (2, ,c2) L,tw Thus we need to study the Joyce-Song invariants JS L (X) of the local K3 gerbe for S S. The 2, ,c2 Ñ Joyce-Song twisted invariants count Gieseker semistable X-twisted sheaves on X. And also from Proposition 6.7, the Behrend function is alway 1. Note that the proof of Proposition 6.7 in [56] only ´ used the local d-critical scheme structure of the moduli space, does not depend on the gerbe structure 0 S S. The result is true for the moduli space of S-twisted sheaves on S. So since ( 1)h (KS) = Ñ L,tw ´ 1, the sign cancels and we need to count the Joyce-Song twisted invariants JS L (X) in the Euler ´ 2, ,c2 characteristic level. L Next we use a gerby version of Toda’s result in [60] to express JS ,tw (X) as the Euler characteristic 2,L,c2 Mss,tw L of the moduli stack S/κ(2, , c2) of S-twisted semistable sheaves on S. Toda’s method in [60] works for category with stability conditions. We will write down more details of Toda’s method for the category of S-twisted sheaves in future work. Since our α = (2, L, c2) can not split, it is always true, see [60, Formula (7)]. And then from Proposition 6.9 this moduli stack is isomorphic to Mss,γ O 1 O the moduli stack S/κ(2, , c2) of γ-twisted semistable sheaves on S of rank 2, determinant and 1 L second Chern class c2, where γ = . From [64], it is deformation equivalent to the Hilbert scheme 1 ´ Hilb2c2 3(S) of points on S. The ﬁrst vanishing is trivial since if c = 0, then E = L O a contradiction with the non-splitting 2 ‘ condition in Proposition 6.10. The second vanishing is from the negative number 2 3 = 1 and ´ 1 ´ ´ ´ Hilb 1(S) = . All other results from the Euler characteristic of Hilb2c2 3(S). H 6.4. Twisted Vafa-Witten invariants-optimal gerbes. In this section we study the twisted Vafa- Witten invariants for optimal µ2-gerbes S S over a K3 surface S. From §2.3, these µ2-gerbes Ñ 2 O˚ correspond to nontrivial order 2 Brauer classes in H (S, S )2´tor, and from Proposition 6.3 and Remark 6.4, there are totally 222 2ρ(S) ´ number of nontrivial order 2 Brauer classes. Let S S be a µ -optimal gerbe over the K3 surface S. Let [S] H2(S, O˚) be its class, then Ñ 2 P S tor [S] represents a class in H1(S, PGL ) since Br1(S) = Br(S) and H2(S, O˚) H1(S, PGL ). We 2 S 2´tor – 2 denote by p : P S Ñ the corresponding projective P1-bundle over S, which is the Brauer-Severi variety. Then from Theorem 3.19: Ms,tw O Proposition 6.13. The moduli stack S/κ(2, , c2) of stable S-twisted Higgs sheaves on S is a µ2-gerbe P,G O over the moduli stack MS/κ(2, , c2) of the G-twisted semistable Higgs sheaves on S. 40 YUNFENG JIANG

We explain the vector bundle G on P and its twisted moduli of Yoshioka. From §3.3, the vector bundle G over P is defined by the Euler sequence 0 O G T 0 Ñ P Ñ Ñ P/S Ñ of the projective bundle P S. Ñ 6.4.1. Integral structure on H˚(S, Q) following Huybrechts-Stellari. The order [S] = 2 in H2(S, O˚) | | S tor is the index ind(S) of the µ2-gerbe S. Therefore 2 is the minimal rank on S such that there exists a rank 2 S-twisted locally free sheaf E on the generic scheme S. Recall that , the Mukai pairing on x y H˚(S, Z) as in Definition 6.1. Recall that in §3.3.1 we defined P-sheaves following Yoshioka. Definition 6.14. For a P-sheaf E, define a Mukai vector of E as: _ Ch(Rp˚(E G )) v (E) := b ?td =(rk, ζ, b) H˚(S, Q), G _ S Ch(Rp˚(G G )) P b where p˚(ζ) = c (E) rk(E) c1(aG) , b Q. 1 ´ rk(G) P One can check that v (E ), v (E ) = χ(E , E ). x G 1 G 2 y ´ 1 2 We define for ξ H2(S, Z), an injective homomorphism: P ˚ ˚ T ξ : H (S, Z) H (S, Q) ´ 2 ÝÑ by ´ ξ x e 2 x ÞÑ ¨ and T ξ preserves the bilinear form , . The following result is from [65, Lemma 3.3]. ´ 2 x y Proposition 6.15. ([65, Lemma 3.3]) Let ξ H2(S, Z) be a representation of ω(G) H2(S, µ ), where P P 2 rk(G) = 2. Set ξ (rk(E), D, a) := e 2 v (E). ¨ G Then (rk(E), D, a) H˚(S, Z) and D mod 2 = ω(E). P In [18], Huybrechts and Stellari defined a weight 2 Hodge structure on the lattice (H˚(S, Z), , ) x y as: 2,0 ˚ 1 ´1 2,0 H (H (S, Z) Aκ)) := T ξ (H (S)); b ´ 2 $H1,1(H˚(S, Z) A1)) := T´1 ( 2 Hp,p(S)); κ ´ ξ p=0 ’ b 2 ‘ &’ 0,2 ˚ 1 ´1 0,2 H (H (S, Z) Aκ)) := T ξ (H (S)). b ´ 2 and this polarized Hodge’ structure is denoted by %’ ξ H˚(S, Z), , , . x y ´ 2 From [65, Lemma 3.4], this Hodge structure H˚(S, Z), , , ξ depends only on the Brauer class x y ´ 2 o([ξ mod 2]), where o(ξ) is the image under the map o : H2(S, µ) H2(S, O˚). 2 Ñ S Definition 6.16. For the projective bundle P S and G the locally free P-sheaf. Let ξ H2(S, Z) be a lifting Ñ P of ω(G) H2(S, µ ), where rk(G) = 2. P 2 (1) Define an integral Hodge structure of H˚(S, Q) as:

˚ ξ T ξ H (S, Z), , , . ´ 2 x y ´ 2 ˚ (2) v = (rk, ζ, b) is a Mukai vector if v T ξ (H (S, Z)) adn ζ Pic(S) Q. Moreover, if v is P ´ 2 P b ˚ primitive in T ξ (H (S, Z)), then v is primitive. ´ 2 COUNTING TWISTED SHEAVES AND S-DUALITY 41

MP,G MP,G In [65], the moduli stack H,ss(v) (or H,s (v)) of G-twisted semistable (or stable) P-sheaves E P,G P,G with vG(E) = v is deﬁned. Let MH,ss(v) (or MH,s (v)) be its coarse moduli space. The moduli stack N P,G N P,G H,ss(v) (or H,s (v)) of G-twisted semistable (or stable) Higgs P-sheaves (E, φ) with vG(E) = v is similarly deﬁned, where E is a G-twisted P-sheaf and

φ : p˚E p˚(E) KS Ñ b _ P,G is the Higgs field. This makes sense since p˚(E L ) is a G-twisted sheaf on S. Let NH,ss(v) (or P,G b NH,s (v)) be its coarse moduli space. From [65, Theorem 2.1], P,G Theorem 6.17. The coarse moduli space NH,s (v) exists, and it is a quasi-projective scheme. Moreover, the natural map N s,tw (v) NP,G(v) is a µ -gerbe. S/κ Ñ H,s 2 N P,G Proof. The moduli space H,s (v) is actually isomorphic to the the coarse moduli space of the moduli stack of µ -gerbe S-twisted Higgs sheaves on S, since the µ -gerbe S has class [S] H2(S, O˚) is 2 2 P S exactly the twisting gerbe in [65]. Therefore the coarse moduli space exists and it is the underlying N s,tw space of the µ2-gerbe, which is isomorphic to S/κ(v), the moduli stack of S-twisted Higgs sheaves (E, φ) with vG(E) = v. Now we can calculate the twisted Vafa-Witten invariants: Proposition 6.18. Let S S be an optimal µ -gerbe over a K3 surface S. Fixing a Mukai vector v = Ñ 2 (2, ζ, b) H˚(S, Q) of rank two. Then the twisted Vafa-Witten invariant P 1 vwtw(S) = VWtw(S) = χ(MP,G (v)). v v 2 H,ss Proof. Since S S is an optimal gerbe, the minimal rank of S-twisted locally free sheaf over generic Ñ scheme S is 2. Thus any rank 2 S-twisted torsion free sheaf is stable. Therefore for any S-twisted N s,tw semistable Higgs sheaf (E, φ), E is stable. Hence in this case, the moduli stack S/κ (v) is an affine Ms,tw cone over the moduli stack S/κ(v) of S-twisted stable sheaves. Since from standard obstruction theory 2 Ext (E, E)0 = 0, Ms,tw N s,tw this moduli stack S/κ(v) is smooth and S/κ(v) is an affine bundle. Thus the twisted Vafa-Witten invariants tw Ms,tw vwv (S) = χ( S/κ(v)) 0 due to the Behrend function value 1, and there is a 1 sign for h (KS). From Theorem 6.17, ´ ´ Ms,tw (v) MP,G (v) is a µ -gerbe, we get S/κ Ñ H,ss 2 1 vwtw(S) = χ(MP,G (v)). v 2 H,ss

Corollary 6.19. Let S S be an optimal µ -gerbe over S. Let v =(2,0, k ) be aMukai vectorfor H˚(S, Q). Ñ 2 ´ 2 Then 1 vwtw(S) = χ(Hilbk+1(S)). v 2 P,G Proof. From [65, Theorem 3.16], the moduli space MH,ss(v) of G-twisted stable sheaves on S with O Mukai vector v = (2,0, b = k ) is deformation equivalent to the moduli space MS, S (v1) of G- ´ ´ 2 S/κ twisted semistable sheaves on S with Mukai vector v1 =(2, D, c) such that 2 2b = D2 2 2c. O ¨ ´ ¨ 1 1 2 S, S 1 2 (v ) +1 1 1 2 And the moduli space MS/κ (v ) is isomorphic to the Hilbert scheme Hilb (S), where 2 (v ) + 1 = 2b + 1. 42 YUNFENG JIANG

In order to get all the positive integers 2b + 1, we have to choose D’s such that D2 = 2 or D2 = 0. ´ When D2 = 2, we get 2b = 1,1,3, , all the odd integers; and when D2 = 0, we get 2b = ´ ´ ¨ ¨ ¨ 0,2,4, , all the even integers. From Proposition 6.18, ¨ ¨ ¨ 1 vwtw = VWtw = χ(MP,G (v)). v v 2 H,ss Thus the result follows.

6.5. Proof of S-duality formula in rank two. In this section we prove the S-duality conjecture Conjecture (1.1.2) for K3 surfaces ﬁrst in rank 2, and then talk about the higher ranks. Let S be a smooth projective K3 surface. Following Conjecture 5.14, and ﬁxing the data (2, O, c2), we consider the partition function

2πig¨O 2 Z(S,SU(2)/Z2; q) = e Z2,Lg (Sg, q), 2 gPHÿ(S,µ2) where tw c Z L (S , q) = vw (S )q 2 2, g g 2,Lg,c2 g c ÿ2 is the partition function of the twisted Vafa-Witten invariants for the µ2-gerbe Sg corresponding to g. 2πig¨O Note that e 2 = 1. We let

Lg is the corresponding deﬁning line bundle for Sg if Sg essentially trivial; #Lg = O if Sg is an optimal gerbe. Theorem 6.20. Let S be a smooth projective K3 surface with Picard number ρ(S). Then

1 1 ρ(S)´ 1 Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 2 1 G( q 2 ) . 2 4 ¨ ¨ ´ ρ(S) Proof. Since the Picard number of S is ρ(S), there are 2 number of essentially trivial µ2-gerbes S S over S. Let S S be the trivial µ -gerbe. Also H2(S, µ ) = 222, and from Remark 6.4, there Ñ 0 Ñ 2 | 2 | are totally 222 2ρ(S) ´ number of nontrivial order 2 Brauer classes, i.e., 222 2ρ(S) number of optimal µ -gerbes. ´ 2 First for the trivial µ -gerbe S S, from Proposition 6.8 and calculations in [56, §5.1], we have: 2 0 Ñ k Z2,O(S0, q) = vw2,0,k(S0)q ÿk 1 = χ(Hilb4k´3(S))q2k + χ(Hilb4k´1(S))q2k+1 + χ(Hilbk(S))q2k 4 ÿk ÿk ÿk 1 1 1 1 = q2G(q2) + q2 G(q 2 ) + G( q 2 ) . 4 2 ´ Let S S be a non-trivial essentially trivial µ -gerbe over S, which is given by the line bundle g Ñ 2 L Pic(S). Then from Proposition 6.12, we have: g P tw k Z L (S , q) = vw (S )q 2, g g 2,Lg,k g ÿk = χ(Hilb4k´3(S))q2k + χ(Hilb4k´1(S))q2k+1 ÿk ÿk 1 1 1 = q2 G(q 2 ) + G( q 2 ) . 2 ´ COUNTING TWISTED SHEAVES AND S-DUALITY 43

Let S S be an optimal µ -gerbe over S. Then from Proposition 6.18 and Corollary 6.19, and g Ñ 2 note that in the Mukai vector, v = (2,0, b = k ), we have b = c + 2 and here we understand ´ ´ 2 ´ ´ 2 that c 1 Z, and the extra 2 comes form the fundamental class 2 ω, we have 2 P 2 ¨ c2 = b + 2 for k Z. Thus we calculate P tw c Z O(S , q) = vw (S )q 2 2, g 2,O,c2 g c ÿ2 tw c2 = vwv (Sg)q c ÿ2 1 k+1 k = χ(Hilb (S))q 2 q2 2 ¨ k kPZ 2 : ÿě´1 1 k k 3 = χ(Hilb (S))q 2 q 2 2 ¨ k kPZ 2 : ÿ ě0 1 1 = q2G(q 2 ). 2 Therefore

Z(S,SU(2)/Z2; q)

= Z2,Lg (Sg, q) 2 gPHÿ(S,µ2) 1 2 2 ρ(S) 1 2 1 1 22 ρ(S) 1 2 1 = q G(q ) + 2 q G(q 2 ) + G( q 2 ) +(2 2 ) q G(q 2 ) 4 ¨ 2 ´ ´ ¨ 2 1 1 ρ(S)´ 1 = q2G(q2) + q2 221G(q 2 ) + 2 1G( q 2 ) . 4 ´

Corollary 6.21. Let S be a smooth projective K3 surface with Picard number ρ(S) = 11. Then 1 1 1 Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) . 2 4 ¨ ¨ ´ Remark 6.22. The formula in Corollary 6.21 is predicted by Vafa-Witten in [61, Formula (4.18)] for any complex K3 surface S. But our result Corollary 6.21 depends on the complex structure of the K3 surface S (it depends on the Picard number). The reason is that we sum all the µ -gerbes S S and the type of µ -gerbes 2 Ñ 2 on S depends on the Picard number. Recall that a K3 surface S over characteristic p 0 is called supersingular if its Picard rank is ρ(S) = ą 22. Then if all the arguments in this paper work for positive characteristic, we get the following result: Corollary 6.23. Let S be a supersingular K3 surface over positive characteristic p 0. Then if all the results ą in the above section work over positive characteristic, we have

1 1 1 Z(S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 221 G( q 2 ) . 2 4 ¨ ¨ ´ SU(2)/Z2 Remark 6.24. We deﬁne vwα (S) to be the Vafa-Witten invariants for the SU(2)/Z2-theory for the surface S. In (1.4.6) we calculate some invariants

SU(2)/Z2 1 SU(2)/Z2 vw2,0 (S) = , vw 3 (S) = 2096128. 4 2, 2 We calculate the coefﬁcient of q2. It is (211 24)+(221 210) 24 + 6 = 50356224 + 6 = 50356230. ˆ ´ ˆ 44 YUNFENG JIANG

Let us explian how to get this invariant. Let S S be a trivial µ -gerbe, then from [56, §5], 0 Ñ 2 1 vwtw (S ) = χ(Hilb1(S)) + χ(Hilb1(S)) = 6 + 24 = 30. 2,0,2 0 4

Let S S be a nontrivial essentially trivial µ -gerbe, then g Ñ 2

tw (S ) = ( 1( )) = vw2,Lg,2 g χ Hilb S 24.

Let S S be an optimal µ -gerbe, then g Ñ 2 1 1 vwtw (S ) = χ(Hilb1(S)) = 24. 2,0,2 g 2 ¨ 2 ¨

Then we have

( ) Z 1 vwSU 2 / 2 (S) = 6 +(211 24) + (222 211) 24 = 50356230. 2,0,2 ˆ 2 ´ ˆ From the calculation in Theorem 6.20, all the fractional degrees of q come form the twisted Vafa-Witten SU(2)/Z2 invariants of optimal µ2-gerbes Sg S. We calculated vw 3 (S) and Ñ 2, 2

SU(2)/Z2 1 22 11 vw 5 (S) = (2 2 ) 324 = 679145472 2, 2 2 ´ ˆ where χ(Hilb2(S)) = 324 is the Euler number of the Hilbert scheme of 2-points on S and

SU(2)/Z2 1 22 11 vw 7 (S) = (2 2 ) 25650 = 53765683200 2, 2 2 ´ ˆ where χ(Hilb4(S)) = 25650 is the Euler number of the Hilbert scheme of 4-points on S. These numbers match the expansion in (1.4.6).

6.6. Vafa-Witten’s S-duality conjecture for K3 surfaces. Vafa-Witten actually predicted the formula in [61, Formula 4.18] for any complex K3 surface S, which does not depend on the complex structure of S. We modify Conjecture 5.14 a bit for K3 surfaces and show that trivial and optimal µ2-gerbes on S give Vafa-Witten’s formula. We follow the notations in §5.5. We fix a smooth projective K3 surface S. The µ -gerbes S S are classified by H2(S, µ ). Let 2 Ñ 2 S S be the trivial µ -gerbe; S S be a nontrivial essentially trivial µ -gerbe corresponding to 0 Ñ 2 ess Ñ 2 a line bundle L ; and S S be an optimal µ -gerbe. g opt Ñ 2 Let us first give a more detailed explanation of Vafa-Witten [61, Page 55]. Vafa-Witten actually sum over the topological data for the PGL2-bundles or Higgs sheaves (E, φ) with first Chern class g = c (E) H2(S, Z)/2 H2(S, Z). From Page 55 in [61], the square g2 satisfies g2 = c (E)2 0,or 2 1 P ¨ 1 ” mod 4. They call g even if g2 0 mod 4; and odd if g2 2 mod 4. There are three type of partition ” ” functions on a K3 surface S corresponding to g = 0, even but not zero and odd. Let n0, neven, nodd be the number of values of g that are respectively, trivial, even but nontrivial, and odd. Then

n0 = 1; 22 11 n = 2 +2 1; $ even 2 ´ ’ 222´211 &nodd = 2 . %’ COUNTING TWISTED SHEAVES AND S-DUALITY 45

From our formula in Corollary 6.21 we modify the calculation: (6.6.1)

11 22 11 Z(S,SU(2)/Z ; q) = Z O(S , q)+(2 1)Z L (S , q)+(2 2 )Z O(S , q) 2 2, 0 ´ r, g ess ´ 2, opt 1 1 1 1 1 1 1 = q2G(q2) + q2 G(q 2 ) + G( q 2 ) +(211 1) q2 G(q 2 ) + G( q 2 ) 4 2 ´ ´ 2 ´ 1 1 +(222 211) q2 G(q 2 ) ´ 2 22 11 1 1 1 1 2 2 1 1 1 = q2G(q2) + q2 G(q 2 ) + G( q 2 ) + (211 1) + ´ q2 G(q 2 ) + G( q 2 ) 4 2 ´ ´ 2 2 ´ h 22 11 i 1 1 2 2 1 1 1 +(222 211) q2 G(q 2 ) ´ q2 G(q 2 ) + G( q 2 ) ´ 2 ´ 2 2 ´ 22 11 1 1 1 1 2 + 2 1 1 1 = q2G(q2) + q2 G(q 2 ) + G( q 2 ) + 1 q2 G(q 2 ) + G( q 2 ) 4 2 ´ 2 ´ 2 ´ h 22 11 i 2 2 1 1 1 + ´ q2 G(q 2 ) G( q 2 ) 2 2 ´ ´ = Z0(q) + nevenZess(q) + noddZodd(q) where we let 1 1 1 1 Z (q) = q2G(q2) + q2 G(q 2 ) + G( q 2 ) ; 0 4 2 ´ 1 1 1 Z (q) = q2 G(q 2 ) + G( q 2 ) ess 2 ´ and 1 2 1 1 Z (q) = q G(q 2 ) G( q 2 ) odd 2 ´ ´ which are the formula in [61, Formula (4.11), (4.12)].

From Proposition 6.8 and Proposition 6.12, we have Z2,O(S0, q) = Z0(q) and Z2,Lg (Sess, q) = Z (q). For optimal µ -gerbes S S, note that for a Mukai vector v =(2,0, b = k ), the second ess 2 opt Ñ ´ ´ 2 Chern classes c2 = 2 + b take values in half integers, we have the following result Lemma 6.25. Deﬁne 2 tw 2c2 c2 Z2,O(Sopt, ( 1) q) := vwv (Sopt)( 1) q . ´ ¨ c ´ ÿ2 Then we have 2 1 2 1 Z O(S , ( 1) q) = q G( q 2 ). 2, opt ´ ¨ 2 ´ Proof. From the calculation in the proof of Theorem 6.20,

2 tw 2c c Z O(S , ( 1) q) = vw (S )( 1) 2q 2 2, opt 2,O,c2 opt ´ ¨ c ´ ÿ2 1 k+1 k+ k = χ(Hilb (S))( 1) 4 q 2 q2 2 ´ ¨ ¨ k kPZ 2 : ÿě´1 1 k k+ k 3 = χ(Hilb (S))( 1) 3 q 2 q 2 2 ´ ¨ ¨ k kPZ 2 : ÿ ě0 1 1 = q2G( q 2 ). 2 ´ 46 YUNFENG JIANG

There are two types of non-zero even classes g H2(S, µ ). The first type consists of classes g P 2 that are algebraic classes (i.e. (1,1)-classes) and the number of these classes is n1 := 2ρ(S) 1. even ´ The second type consists of classes g that are non-algebraic classes and the number of these classes is n2 := n (2ρ(S) 1). even even ´ ´ Definition 6.26. For the type of even classes g, we define

2 1 2 1 1 Z (q) := Z O(S , q) + Z O(S , ( 1) q) = q G(q 2 ) + G( q 2 ) . even 2, opt 2, opt ´ 2 ´ The partition function Zodd(q) is obtained from optimal µ2-gerbes by:

1 2 1 1 2πi 1 2 Z (q) = q G(q 2 ) G( q 2 ) = Z O(S , q)+( 1) 2 Z O(S , ( 1) q). odd 2 ´ ´ 2, opt ´ ¨ 2, opt ´ Let L Pic(S) be a nontrivial line bundle such that it deﬁnes a nontrivial essentially trivial µ - g P 2 gerbe on S. Then c (L) = g mod 2 and g 0. We have 1 ‰

Proposition 6.27. Let Z2,Lg (S, q) be the generating function of Tanaka-Thomas Vafa-Witten invariants for S with rank two and with determinant Lg. Then

˚ Z2,Lg (S, q) = Z2,p Lg (Sess, q) = Zess(q) = Zeven(q), where p : S S is the map to its coarse moduli space. Moreover the formula Z L (S, q) only depends on ess Ñ 2, g the ﬁrst Chern class c H2(S, Z) such that 0 g = c mod 2. 1 P ‰ 1 Proof. The proof of Proposition 6.10 and Proposition 6.12 hold for semistable Higgs shaves (E, φ) on the K3 surface S such that det(E) = Lg. Thus we get the same formula for the generating function in the calculation in the proof of Theorem 6.20. The last equality is from deﬁnition.

2 Remark 6.28. Let L Pic(S) be a line bundle such that its image in H (S, µ2) is nonzero. For the optimal P tw µ2-gerbe Sopt over S, from [65, Theorem 3.16], the moduli stack M (Sopt) of stable twisted sheaves (2,L,c2) Mtw with ﬁxed determinant L Pic(S) is isomorphic to the moduli stack ( O 1 )(Sopt) with trivial determinant P 2, ,c2 1 for suitable c2. Thus their corresponding stable twisted Higgs sheaves are also deformation equivalent, which shows that Deﬁnition 6.26 is consistent for the two type of even classes. Remark 6.29. From Proposition 6.12, for a nontrivial essentially trivial µ -gerbe S S corresponding to 2 ess Ñ a line bundle L, we have

tw c Z L(S , q) = VW (S )q 2 2, ess 2,L,c2 ess c ÿ2 = χ(Hilb4k´3(S))q2k + χ(Hilb4k´1(S))q2k+1 kÿě1 kÿě1 1 1 1 = q2 G(q 2 ) + G( q 2 ) 2 ´ From the proof of Theorem 6.20, and Lemma 6.25, the above formula is equal to 2 Z O(S , q) + Z O(S , ( 1) q) 2, opt 2, opt ´ ¨ 1 k k 3 1 k k+ k 3 = χ(Hilb (S))q 2 q 2 + χ(Hilb (S))( 1) 3 q 2 q 2 2 ¨ 2 ´ ¨ ¨ k kPZ k kPZ 2 : ÿ ě0 2 : ÿ ě0 1 1 1 = q2 G(q 2 ) + q2G( q 2 ) . 2 ´ Therefore the moduli stack we choose for even classes does not depend on the algebraic or non-algebraic classes, which is all deformation equivalent to the Hilbert scheme Hilbk(S) of points on S. COUNTING TWISTED SHEAVES AND S-DUALITY 47

Theorem 6.30. Let S be a smooth complex K3 surface, and deﬁne 1 Z2,O(S,SU(2)/Z2; q) := Z0(q) + nevenZeven(q) + noddZodd(q).

We call it the partition function of SU(2)/Z2-Vafa-Witten invariants, and we have:

1 1 1 1 Z (S,SU(2)/Z ; q) = q2G(q2) + q2 221 G(q 2 ) + 210 G( q 2 ) . 2 4 ¨ ¨ ´ This proves the prediction of Vafa-Witten in [61, §4] for the gauge group SU(2)/Z2 and the S-duality conjecture (1.1.2). 1 Proof. The proof of the formula Z (S,SU(2)/Z2; q) is given in (6.6.1), since 1 1 1 Z (q) = q2 G(q 2 ) + G( q 2 ) . even 2 ´ Since for a complex K3 surface S, the Picard number ρ(S) satisfies 0 ρ(S) 20. We have the ď ď inequality n 2ρ(S) = n1 + 1. Therefore there definitely exist optimal µ -gerbes on S such that even ą even 2 their corresponding classes g H2(S, µ ) have two parts. One part corresponds to even classes g that P 2 are not algebraic, and the other corresponds to odd classes. Then from Definition 6.26 and Remark 1 6.28 it is not hard to see that the formula Z (S,SU(2)/Z2; q) is independent of the complex structure on S. Remark 6.31. For the even class g H2(S, Z ) such that g 0, we define P 2 ‰ Z1 (S,SU(2)/Z ; q) := Z (q)+(210 1)Z (q) 210Z (q). 2,even 2 0 ´ ess ´ odd For odd classes g H2(S, Z ), we define P 2 Z1 (S,SU(2)/Z ; q) := Z (q)+( 210 1)Z (q) + 210Z (q). 2,odd 2 0 ´ ´ ess odd Then from [61, (4.14), (4.15)] under the S-transformation τ 1 we have: ÞÑ ´ τ τ w/2 Z (q) 2´11 Z1 (S,SU(2)/Z ; q); even ÞÑ i 2,even 2 and τ w/2 Z (q) 2´11 Z1 (S,SU(2)/Z ; q). odd ÞÑ i 2,odd 2 Remark 6.32. If the K3 surface S is projective, then the Picard number satisfies the condition 1 ρ(S) 20. ˚ ď ď In this case we can define Zess(q) = Z2,p Lg (Sess, q) just using one nontrivial essentially trivial µ2-gerbe Sess S, since there exists a nontrivial line bundle Lg Pic(S) such that c1(Lg) g mod 2 and 0 g 2 Ñ P ” ‰ P H (S, µ2). 6.7. Discussion on higher rank case. Let S be a smooth K3 surface with Picard number ρ(S). Let us first fix to the case the rank r is a prime integer. Then like r = 2, we have similar results for essentially trivial µ -gerbes S S. For the trivial µ -gerbe S = [S/µ ] S, the twisted Vafa-Witten invariants r Ñ r r Ñ are the same as the Vafa-Witten invariants in [56]. For non-trivial essentially trivial µ -gerbes S S, we have: r Ñ Proposition 6.33. Let (E, φ) be a S-twisted semistable rank r Higgs sheaf with class (r, L, c2). Then the torsion free sheaf E can not split into sum of subbundles. Proof. The proof is similar to Proposition 6.10. The reason is that if E can be split into E = E E 1 ‘ 2 such that each Ei is of higher rank less than r, the determinant is det(E) = L, which implies that s det(E) = det(E E ). Either E or E is nontrivial, and det(E ) = L r for some 1 s r. The basic 1 ‘ 2 1 2 i ď ď reason is that r do not have nontrivial split. Thus c1(E) = c1(E1) + c1(E2) and after modulo mod r we get that c (E) 0 mod r, which is a contradiction. 1 ” The following is a similar result as in Proposition 6.12, we omit the proof. 48 YUNFENG JIANG

Proposition 6.34. Let S S be a non-trivial essentially trivial µ -gerbe over the K3 surface S corresponding Ñ r to the line bundle L Pic(S). Then we have P 2 2 tw r ¨k´(r ´1) vwr,L,rk(S) = χ(Hilb (S)); 2 2 tw r ¨k´(r ´r) $vwr,L,rk+1(S) = χ(Hilb (S)); ’ ’. . . . ’. . . . & 2 tw S r ¨k´1 vwr,L,rk+(r´1)( ) = χ(Hilb (S)), ’ n ’ where χ(Hilb (S)) is the topological%’ Euler number of the Hilbert scheme of n points on S. 2 22 Since r is a prime number, H (S, µr) = Zr , all other non essentially trivial µr-gerbes on S is an optimal µ -gerbe. Let S S be an optimal µ -gerbe over S. All of the arguments in §6.4 works for r Ñ r this µr-gerbe S, since we only count rank r S-twisted Higgs sheaves, and the minimal rank of locally free S-twisted sheaf is r, hence must be stable. Especially we have a similar result comparing with Corollary 6.19:

Corollary 6.35. Let S S be an optimal µ -gerbe over S for a prime number r. Let v = (r,0, k ) be a Ñ r ´ r Mukai vector for H˚(S, Q). Then 1 vwtw(S) = χ(Hilbk(S)). v r Combining with Proposition 6.34 and Proposition 6.35, we have the following result: Theorem 6.36. Let S be a smooth projective K3 surface with Picard number ρ(S). Then

r´1 1 r r r 21 1 ρ(S)´1 2πij 1 Z(S,SU(r)/Zr; q) = q G(q ) + q r G(q r ) + r G e r q r . r2    = jÿ1    Proof. We do similar calculation as in Theorem 6.20. The Picard number of S is ρ(S), there are rρ(S) number of essentially trivial µ -gerbes S S over S. Let S S be the trivial µ -gerbe. Also r Ñ 0 Ñ 2 H2(S, µ ) = r22, and there are totally | r | r22 rρ(S) ´ number of nontrivial order r Brauer classes, i.e., these number of optimal µr-gerbes. For the trivial µ -gerbe S S, from Proposition 6.8 and calculations in [56, §5.1], we have: r 0 Ñ k Zr,O(S0, q) = vwr,0,k(S0)q ÿk r´1 1 r r 1 r 2πij 1 = q G(q ) + q G e r q r . r2 r   jÿ=0   And this is the result in [56, §5]. Let S S be a non-trivial essentially trivial µ -gerbe over S, which is given by the line bundle g Ñ r L Pic(S). Then from Proposition 6.34, we have: g P r´1 r´1 2πij 1 tw k tw k 1 r Z L (Sg, q) = vw L (Sg)q = vw L (Sg) q = q G e r q r . r, g r, g,k  r, g,rk+j  r   ÿk ÿk jÿ=0 jÿ=0     Let S S be an optimal µ -gerbe over S. Then from Corollary 6.35, and in the Mukai vector, g Ñ r v = (r,0, b = k ), we have b = c + r and here we understand that c 1 Z, here the extra r ´ ´ r ´ ´ 2 2 P r comes form the fundamental class r ω, we have ¨ c2 = b + r COUNTING TWISTED SHEAVES AND S-DUALITY 49 for k Z . Thus we calculate: P ě0 tw c tw c Z O(S , q) = vw (S )q 2 = vw (S )q 2 r, g r,O,c2 g v g c c ÿ2 ÿ2 1 k+1 k r = χ(Hilb (S))q r q r ¨ k PZ r :k ÿě´1 2 1 k k r ´1 = χ(Hilb (S))q r q r r ¨ k kPZ r : ÿ ě0 1 r 1 = q G(q r ). r Therefore

Z(S,SU(r)/Zr; q) = Zr,Lg (Sg, q) 2 gPHÿ(S,µr) r´1 1 ( ) 1 2πij 1 ( ) 1 1 = qrG(qr) + rρ S qr G e r q r +(r22 rρ S ) qrG(q r ) r2 ¨ r   ´ ¨ r = jÿ0  r´1 1 r r r 1 ρ(S)´ 2πij 1 = q G(q ) + q r21G(q r ) + r 1 G e r q r . r2    = jÿ1   

We also provide a proof of Vafa-Witten’s conjecture for prime ranks. Comparing with Lemma 6.25, for any optimal µ -gerbe S S, we have r opt Ñ πi m Lemma 6.37. For any r-th root of unity e2 r (0 m r 1), deﬁne ď ă ´ πim m 2 r tw 2πi rc c Z O(S , (e r ) q) := vw (S )(e r ) 2 q 2 . r, opt ¨ v opt P 1 Z c2ÿr Then we have 2πim r 1 r 2πim 1 Z O(S , (e r ) q) = q G(e r q r ). r, opt ¨ r Proof. From the calculation for optimal gerbes in Theorem 6.36, the proof is similar to Lemma 6.25.

Let Zr,0(q) denote the Tanaka-Thomas partition function for S with rank r and trivial determinant O. Theorem 6.38. Let S be a smooth complex K3 surface. For a prime positive integer r, we deﬁne

r´1 ´ 1 πi r 1 mg2 2πim r Z O(S,SU(r)/Z ; q) := Z (q) + e r Z O(S , (e r ) q) r, r r,0 r, opt ¨ 2 m=0 0‰gPÿH (S,µr) ÿ We call it the partition function of SU(r)/Zr-Vafa-Witten invariants, and we have: r´1 1 1 r r r 21 1 10 2πim 1 Z (S,SU(r)/Zr; q) = q G(q ) + q r G(q r ) + r G e r q r . r2 = !! mÿ1 This generalizes the prediction of Vafa-Witten in [61, §4] to the gauge group SU(r)/Zr and proves Formula (4.11) in [39]. 50 YUNFENG JIANG

Proof. First Tanaka-Thomas partition function is: r´1 1 r r 1 r 2πim 1 Z (q) = q G(q ) + q G e r q r . r,0 r2 r m=0 ! ÿ 2πim 1 2πim 1 From Lemma 6.37, we always have Z O(S , (e r )r q) = qrG(e r q r ) for any 0 m r 1. r, opt ¨ r ď ă ´ Now from [39, Formula (A.15)], and note that S is a K3 surface, when m = 0, in the sum of the 1 deﬁnition of Z1 (S,SU(r)/Z ; q) there totally r22 terms of 1 qrG(q q ), and when m 0, r,O r r ‰ πi r´1 mg2 e r =(ǫ(m))22r11 2 gPHÿ(S,µr) m/2 (m+r)/2 a where ǫ(m) = r if m is even; and ǫ(m) = r if m is odd. The number r is the Legendre symbols which is 1 if a( mod r) is a perfect square, and 1 otherwise. Thus (ǫ(m))22 = 1. Then the ´ formula in the theorem follows from [39, Formula (A.15)]. Remark 6.39. The result in Theorem 6.38 is also independent of the complex structure of S. For the general higher rank case, we need a multiple cover formula for the twisted Joyce-Song invariants introduced by Toda [60] for µr-gerbes, see [29]. .

REFERENCES [1] M. Artin, Brauer-Severi varieties, in: Brauer groups in ring theory and algebraic geometry (Wilrijk, 1981), pp. 194-210, Lecture Notes in Math. 917, Springer, Berlin-New York, 1982. [2] M. Artin and A. J. de Jong, Stable orders over surfaces, 2003, preprint. [3] E. Andreini, Moduli spaces of pairs over projective stacks, arXiv:1105.5637. [4] E. Andreini, Y. Jiang and H.-H. Tseng, Gromov-Witten theory of etale gerbes I: the structure of moduli spaces, Journal Differential Geometry. 99 (2015), no. 1, 1-45. arXiv:0907.2087. [5] E. Andreini, Y. Jiang and H.-H. Tseng, Gromov-Witten theory of etale gerbes II: higher genus invariants, arXiv:1101.5996. [6] K. Behrend, Donaldson-Thomas invariants via microlocal geometry, Ann. Math. (2009), Vol. 170, No.3, 1307-1338, math.AG/0507523. [7] K. Behrend and B. Fantechi, The intrinsic normal cone, alg-geom/9601010, Invent. Math. 128 (1997), no. 1, 45-88. [8] K. Behrend and P. Ronagh, The inertia operator on the motivic Hall algebra, arXiv:1612.00372. [9] T. Bridgeland, An introducation to motivic Hall algebras, Adv. Math. 229, no. 1, 102-138 (2012). [10] A. Caldararu, Derived categories of twisted sheaves on Calabi-Yau manifolds, 2000. Thesis, Cornell University. [11] T. Coates, H. Iritani, Y. Jiang and E. Segal, K-theoretical and categorical properties of toric Deligne-Mumford stacks, Pure and Applied Mathematics Quarterly, 11 (2015) No.2, 239-266. arXiv:1410.0027. [12] A. J. de Jong, A result of Gabber, 2003, Preprint. [13] B. Fantechi and L. Gottsche, Riemann-Roch theorems and elliptic genus for virtually smooth schemes, Geom. Top. 14 83- 115, 2010. arXiv:0706.0988. [14] T. Graber and R. Pandharipande, Localization of virtual classes, Invent. Math. 135 487-518, 1999. alg-geom/9708001. [15] A. Gholampour, Y. Jiang, and M. Kool, Sheaves on weighted projective planes and modular forms, Advances in Theoretical and Mathematical Physics, Vol. 21, NO. 6, (2017) 1455-1524, arXiv:1209.3922. [16] L. Göttsche, Modular forms and Donaldson invaraints for 4-manifolds with b+ = 1, Journal of the American Mathematical Society, Vol. 9, No. 3 (1996), 827-843. [17] L. Göttsche and M. Kool, Virtual refinements of the Vafa-Witten formula, arXiv:1703.07196. [18] D. Huybrechts and P. Stellari, Equivalences of twisted K3 surfaces, Math. Ann., 332 (2005), 901-936. [19] D. Huybrechts and S. Schröer, The Brauer group of analytic K3 surfaces, International Mathematics Research Notices, Vol. 2003, Issue 50, (2003), Pages 2687-2698. [20] D. Huybrechts, Lectures on K3 surfaces, Cambridge Studies in Advanced Mathematics, 158. Cambridge University Press, Cambridge, 2016. [21] D. Huybrechts and M. Lehn, The geometry of moduli spaces of sheaves, Aspects of Mathematics, E31, Friedr. Vieweg & Sohn, Braunschweig, 1997. MR MR1450870 (98g:14012). [22] Y. Jiang and R. Thomas, Virtual signed Euler characteristics, Journal of Algebraic Geometry, 26 (2017) 379-397, arXiv:1408.2541. [23] Y. Jiang, Note on MacPherson’s local Euler obstruction, Michigan Mathematical Journal, 68 (2019), 227-250, arXiv:1412.3720. [24] Y. Jiang, and P. Kundu, The Tanaka-Thomas’s Vafa-Witten invariants for surface Deligne-Mumford stacks, arXiv:1903.11477. COUNTING TWISTED SHEAVES AND S-DUALITY 51

[25] Y. Jiang, Donaldson-Thomas invariants of Calabi-Yau orbifolds under flops, Illinois Journal of Math., Volume 62, No 1-4 (2018), 61-97, arXiv:1512.00508. [26] Y. Jiang, The Vafa-Witten invariants for surface Deligne-Mumford stacks and S-duality, Survey for ICCM-2019, arXiv:1909.03067. [27] Y. Jiang, On the construction of moduli stack of projective Higgs bundles over surfaces, preprint, arXiv:1911.00250. [28] Y. Jiang and M. Kool, Twisted sheaves and SU(r)/Zr Vafa-Witten theory, arXiv:2006.10368. [29] Y. Jiang and H.-H. Tseng, Stable pair invariants for K3 gerbes and higher rank S-duality conjecture for K3 surfaces, arXiv:2003.09562. [30] D. Joyce, Configurations in abelian categories III. Stability conditions and identities, Adv. Math., 215 (2007), 153-219. [31] D. Joyce and Y. Song, A theory of generalized Donaldson-Thomas invariants, Memoris of the AMS, 217 (2012), 1-216, arXiv:0810.5645. [32] D. Joyce, Configurations in abelian categories II: Ringel-Hall algebras, Adv. Math. 210 (2007), No.2, 635-706. [33] Y.-H. Kiem and J. Li, Localizing virtual cycles by cosections, Jour. A.M.S. 26 1025-1050, (2013), arXiv:1007.3085. [34] A. A. Klyachko, Vector bundles and torsion free sheaves on the projective plane, preprint, Max Planck Institut fur Mathematik (1991). + [35] D. Kotschick and J. W. Morgan, SO(3)-Invariants for 4-manifolds with b2 = 1 II, J. Differential Geometry, 39 (1994) 433-456. [36] M. Kool, Fixed point loci of moduli spaces of sheaves on toric varieties, Adv. Math., Vol. 227, Iss. 4, (2011), 1700-1755. [37] T. Laarakker, Monopole contributions to refined Vafa-Witten invariants, arXiv:1810.00385. [38] T. Laarakker, Vertical Vafa-Witten invariants, arXiv:1906.01264. [39] J. M. F. Labastida and C. Lozano, The Vafa-Witten theory for gauge group SU(N), arXiv:hep-th/9903172. [40] G. Laumon and L. Moret-Bailly, Champs Algebriques, Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge/A Series of Modern Surveys in Mathematics. [41] J. Li and G. Tian, Virtual moduli cycles and Gromov-Witten invariants of algebraic varieties, J. Amer. Math. Soc., 11, 119- 174, 1998, math.AG/9602007. [42] W. Li and Z. Qin, On blowup formulae for the S-duality conjecture of Vafa and Witten. Invent. Math. 136 (1999), 451-482. [43] M . Lieblich, Moduli of twisted sheaves, Duke Math. J., Vol. 138, No. 1 (2007), 23-118. [44] M . Lieblich, Twisted sheaves and the period-index problem, Compositio. Math., Vol. 144, Iss. 1 (2008) , 1-31. [45] M . Lieblich, Compactified moduli of projective bundles, Algebra Number Theory, Vol. 3 (2009), No. 6, 653-695. [46] K. Matsuki and R. Wentworth, Mumford-Thaddeus principle on the moduli space of vector bundles on an algebraic surface, Internat. J. Math., 8(1):97-148, 1997. [47] D. Maulik, Supersingular K3 surfaces for large primes, Duke Math. J. Vol. 163, No. 13 (2014), 2357-2425. [48] D. Maulik and R. P. Thomas, Sheaf counting on local K3 surfaces, arXiv:1806.02657. [49] J. S. Milne, Etale´ cohomology, Princeton University Press, Princeton, New Jersey. [50] C. Montonen and D. I. Olive, Magnetic Monopoles As Gauge Particles, Phys. Lett. B 72, 117 (1977). [51] G. Moore and E. Witten, Integration over the u-plane in Donaldson theory, Adv. Theor. Math. Phys. 1 (1997), no. 2, pages 298-387. [52] F. Nironi, Moduli Spaces of Semistable Sheaves on Projective Deligne-Mumford Stacks, arXiv:0811.1949. [53] C. Simpson, Moduli of representations of the fundamental group of a smooth projective variety II Publications Mathématiques de lHES´ , Vol. 80 (1994), p. 5-79. [54] Stack Project: https://stacks.math.columbia.edu/download/stacks-morphisms.pdf. [55] Y. Tanaka and R. P. Thomas, Vafa-Witten invariants for projective surfaces I: stable case, J. Alg. Geom. to appear, arXiv.1702.08487. [56] Y. Tanaka and R. P. Thomas, Vafa-Witten invariants for projective surfaces II: semistable case, Pure Appl. Math. Q. 13 (2017), 517-562, Special Issue in Honor of Simon Donaldson, arXiv.1702.08488. [57] X. Tang and H.-H. Tseng, Duality theorems for étale gerbes on orbifolds, Advances in Mathematics 250 (2014), 496-569. [58] R. P. Thomas, A holomorphic Casson invariant for Calabi-Yau 3-folds, and bundles on K3 fibrations, J. Differential Geom., 54, 367-438, 2000. math.AG/9806111. [59] R. P. Thomas, Equivariant K-theory and refined Vafa-Witten invariants, preprint, arXiv:1810.00078. [60] Y. Toda, Stable pairs on local K3 surfaces, Jour. Diff. Geom. 92 (2012), 285-370. [61] C. Vafa and E. Witten, A strong coupling test of S-duality, Nucl. Phys. B 431 3–77, 1994. hep-th/9408074. [62] A. Vistoli, Intersection theory on algebraic stacks and on their moduli spaces, Invent. Math., 97 (1989) 613-670. [63] K. Yoshioka, The Betti numbers of the moduli space of stable sheaves of rank 2 on P2, Kyoto University. [64] K. Yoshioka, Some examples of Mukai’s reflections on K3 surfaces, J. Reine Angew. Math., 515 (1999), 97-123. [65] K. Yoshioka, Moduli spaces of twisted sheaves on a projective variety, Advanced Studies in Pure Mathematics 45, 2006 Moduli Spaces and Arithmetic Geometry (Kyoto, 2004) pp. 1-42. [66] D. Zagier, Nombres de classes et formes modulaires de poids 3/2, C. R. Acad. Sc. Paris, 281A, 883-886 (1975).

DEPARTMENT OF MATHEMATICS,UNIVERSITYOF KANSAS,405SNOW HALL 1460 JAYHAWK BLVD,LAWRENCE KS 66045 USA Email address: [email protected]