Topological Magnetic Crystalline Insulators and Co-Representation

arXiv:1401.6922v1 [cond-mat.mes-hall] 27 Jan 2014 el[,1] ib1] Bi quantum been HgTe BiSb[14], have as 13], such TIs and invariant materials, two wells[9, various TR both in in reversal symmetry pre- confirmed [6–12]. time TR were dimensions with example, system three (TIs) a For insulators in dicted topolog- topological was appear. invariant new it can (TR) symmetries, fi- Recently, a phases certain of ical under edge Hall sample. that 1D quantum (2D) realized the state a at two-dimensional edge In direction nite chiral topolog- one (1D) a along [5]. one-dimensional of propagates dimensions gapless example a two first state, in The the state is states. ical an state edge/surface by cannot Hall metallic characterized quantum that and usually are Topological matter bulk fermions insulating of [1–4]. free principles state of symmetry phases quantum by of classified be type new a hss hrfr,tersac ntplgclphases topological topological on in research dissipation the low electric Therefore, with so flow sup- phases. states, significantly can edge/surface are currents topological It backscatterings for topological that pressed symmetry. a mirror believed by be is protected to insulator [35–37] and crystalline [34] example, confirmed predicted For theoretically experimentally been have symmetry[28– etc. systems crystalline SnTe symmetry[31–33], as other particle-hole such by 30], protected symmetries, sym- be of also TR types can Besides states topological 27]. metry, states”[26, dubbed thus edge/surface are and “helical edge/surface, counter- given a spin-polarization at consist opposite propagating The TIs with invariant branches TR two of of 23–25]. states (STM) measurements[13, edge/surface microscopy metallic transport tunnelling and scanning 19], [20–22] 17, 16, [14, (ARPES) spectroscopy emission including photon methods, angular-resolved experimental different by etc[18], 17], ncnesdmte hsc,atplgclsaeis state topological a physics, matter condensed In 1 oooia antccytlieisltr n co-repre and insulators crystalline magnetic Topological eateto hsc,TePnslai tt University, State Pennsylvania The Physics, of Department ASnmes 32.t 34.f 75.50.Pp 73.43.-f, 73.20.At, topol numbers: PACS for search to way materials. the magnetic pave realistic will in work Our states. surface acltdepiil.Mroe,w hc ieettpso with types systems different the check that find we and Moreover, systems explicitly. calculated C ecntuttocnrt ih-idn oeso topologi of models of tight-binding theory concrete co-representation two cr the construct magnetic We on magn topological based of “topological crystals theory dubbed general magnetic a are present we structures work, magnetic b cr in explicitly and symmetry is phases reversal time symmetry recentl combining reversal Very operation time symmetry operation. which reversal in time structures, the of nature unitary ˆ 4 als ufc ttso iervra nain topologi invariant reversal time of states surface Gapless oe n h ˆ the and model Θ .INTRODUCTION I. 2 Se 3 τ aiyo materials[15– of family oe,i hc oooia ufc ttsadtopologica and states surface topological which in model, Θ u-igZhang Rui-Xing Dtd etme 5 2018) 25, September (Dated: C ˆ 4 Θ, 1 C ˆ n hoXn Liu Chao-Xing and 6 n ˆ and Θ ie h Rsmer operator symmetry TR com- that the operator anti-unitary bines sys- an anti-ferromagnetic possesses usually an tem example, For mag- various but materials. including system, systems, netic invariant breaking TR TR in a exist in also appear topo- only not for anti-unitary However, operators essential states. is surface non-trivial operators logically anti-unitary by induced [39]. TIs e.g. doped gapped, magnetically be of case can and the exists, states in longer surface/edge no topological TRIM at the symmetry degeneracy invariant TR the TR When broken, degeneracy. in is this Dirac states by The protected surface/edge is helical degenerate. TIs the doubly of be must point the TRIM All at (TRIM)”. momenta states invariant “TR as known tor), oec te yT ymty omn h so-called of at the state pairs” one forming momentum, “Kramers’ symmetry, generic the a related TI, TR For are invariant by pairs”. “Kramers’ states TR other a surface/edge each In helical to of level. branches energy two each for generally prtrˆ operator nld niuiayoeaosaekona magnetic as known are that operators groups symmetry anti-unitary In the include crystallography, of [40–42]. study the insulator” topological “anti-ferromagnetic rtrˆ erator at condition pnesfrin rbsn.Kaes hoe 3]tells [38] when theorem that Kramers’ us bosons. or fermions spinless ntr.FraT nain ytm h Roperator TR the system, invariant TR sastifies a For unitary. a oeta plcto neetoiswt o dissi- low with electronics in pation. application potential a has ymtyoeaos Rsmer operator symmetry TR crystalline operators, unitary to Unlike symmetry due 27]. degeneracy 10, 7, double theorem[6, Kramers’ from originates TIs invariant rmti icsin ecnseta h degeneracy the that see can we discussion, this From h als aueo eia desraesae nTR in states edge/surface helical of nature gapless The − k τ hr r oeseilmmnastsyn the satisfying momenta special some are There . ymtyaeal opoetgapless protect to able are symmetry Θ τ a nuaosaepoetdb h anti- the by protected are insulators cal ,ti dawsgnrlzdt magnetic to generalized was idea this y, oe,btteei tl nanti-unitary an still is there but roken, nvriyPr,Pnslai 16802-6300 Pennsylvania Park, University a antccytlieisltr,the insulators, crystalline magnetic cal groups. symmetry crystalline magnetic saln nuaosi ieettpsof types different in insulators ystalline a lola oa to lead also can Θ ˆ Θ saln ymty hs topological These symmetry. ystalline niuiayoeaosi magnetic in operators anti-unitary f ˆ τ gclmgei rsaln insulators crystalline magnetic ogical k tccytlieisltr” nthis In insulators”. crystalline etic 2 thsbe hw htteat-ntr op- anti-unitary the that shown been has It . = = Θ − ˆ − 2 1 k o pnu emosand fermions spinful for 1 + = G − ,adul eeeayappears degeneracy double a 1, ( etto theory sentation G k Z sdgnrt ihteother the with degenerate is stercpoa atc vec- lattice reciprocal the is 2 nainsare invariants l oooia hs,dubbed phase, topological ihatranslation a with Θ ˆ Θ ˆ santi- is Θ 2 ˆ for 1 = 2 symmetry groups, as an extension of the conventional II. MAGNETIC CRYSTALLINE SYMMETRY crystalline symmetry groups. Just like the classification of crystal structures according to crystalline symmetry A. Magnetic point groups and magnetic space groups, different types of magnetic structures are classi- groups fied by magnetic symmetry groups. Therefore, it is natural to ask what types of magnetic symmetry groups, We start from a review of the structure of magnetic as well as their corresponding magnetic structures, can symmetry groups and the so-called “co-representation” support topologically non-trivial phases. theory [38, 43]. Magnetic symmetry groups include both To answer this question, we develop a theory for topo- unitary operations of crystalline symmetry and anti- logical phases in magnetic crystals, dubbed “topologi- unitary symmetry operations due to TR symmetry. Sim- cal magnetic crystalline insulators (TMCIs)”, based on ilar to point groups and space groups, we also have mag- the “co-representation” theory of magnetic groups in this netic point symmetry groups and magnetic space sym- paper [38, 43]. It is well-known that the degeneracy of metry groups. There are in total 122 magnetic point energy states is related to irreducible representations of symmetry groups that can be classified into three types. a crystalline symmetry group in the representation the- Type I groups are just the ordinary point groups, de- ory. However, the conventional representation theory noted as G, so there are 32 of them. Type II groups can not be directly applied to magnetic groups due to are the direct product of an ordinary point group G the existence of anti-unitary operators. For example, with the group E, Θˆ , where E is the identity and Θˆ in a TR invariant system, the double degeneracy due is the TR symmetry.{ } There are also 32 different types to Kramers’ theorem of TR symmetry is regarded as of type II groups. The most interesting groups are the an “additional degeneracy” that is not included in the type III groups, which take the form M = G+AG where conventional representation theory [38]. In order to un- A =ΘR and the crystalline symmetry operation R does derstand this “additional degeneracy”, Wigner first de- not belong to G. There are 58 type III magnetic point veloped the so-called “co-representation” theory [43, 44]. symmetry groups in total. For a type III group, half The co-representation is different from the conventional of the elements are unitary and the other half are anti- representation because the multiplication rule is modi- unitary. Magnetic space groups are constructed by the fied due to anti-unitary operators. Consequently, some product of the translation symmetry group and the mag- useful concepts in the conventional representation the- netic point symmetry group. 1651 magnetic space groups ory, such as characters, can no longer be applied to the can also be categorized into three classes, 230 ordinary co-representation theory. The additional degeneracy of crystallographic space groups (Type I), 320 type II and energy states due to anti-unitary operators can be de- 1191 type III groups. We can further divide the type scribed well in the co-representation theory. Since the Z2 III magnetic symmetry group into two classes: type IIIa topological phases are closely related to the degeneracies where R cannot be chosen to be a pure translation and induced by anti-unitary operators, the co-representation type IIIb where R is just a pure translation. For type 2 provides a natural approach to investigate Z2 topological IIIb, R must be a translation operator, so in this case phases in magnetic structures. the subgroup S = T + Θˆ RT characterizes the so-called In this paper, we will first review magnetic symmetry magnetic Bravais lattice, where T is the translation sym- groups, and then discuss our approach to generalize Z2 metry group. There are 674 of type IIIa and 517 of type topological phases to a magnetic system based on the co- IIIb. representation theory of magnetic symmetry groups. We Before going into a detailed discussion, we will first will consider two concrete tight-binding models for TM- illustrate our notations. We usually denote a magnetic CIs. One is related to the combination of TR symmetry symmetry group by M(G) [43], where M is for the mag- G and four-fold rotation symmetry, dubbed “Cˆ4Θˆ model”, netic group and is for its unitary part. The translation while the other is related to the combination of TR sym- subgroup is denoted as T. Below, we will use t, w, v, ˆ for translation operators, R, S, W, for unitary point··· metry and translation symmetry, dubbed “ˆτΘ model”. ··· We utilize the Wilson loop technique [45, 46] to calcu- symmetry operators and A, B, for anti-unitary opera- ··· ˆ late the corresponding Z2 topological invariant in these tors. TR symmetry is always denoted by Θ. An element in a space group is denoted as S t where S is for the two models. Finally, we will generalize our discussion to { | } other types of magnetic symmetry groups and show that point group symmetry and t is for the translation oper- a Z Z topological magnetic crystalline phase exists ation. 2 × 2 in a system with Cˆ6Θˆ symmetry. The paper is orga- nized as follows: In Sec. II, we will summarize the co- representation theory of magnetic crystalline groups and B. Co-representation theory and degeneracy discuss the principle used to define Z2 topological invariants in magnetic structures. In Sec. III, we discuss two It is well-known that electronic states can serve as the concrete models for Z2 TMCIs, as well as the generaliza- basis for the construction of representations of a symme- tion to other magnetic symmetry groups. Conclusion is try group and degeneracies in electronic band structures drawn in Sec. IV. are directly determined by irreducible representations of 3 the corresponding symmetry group [38]. The irreducible S G commutes with A, Eq. (A4) is simplified as representations can be read out from the character table ∈ ∆(S) 0 of a symmetry group. However, type II and type III mag- D(S)= . (2) netic symmetry groups contain anti-unitary operators, 0 ∆∗(S) which are either a TR operation or a combination of a For any B = AS, Eq. (A5) takes the form TR operation and another unitary operation. This pre- vents the construction of conventional representations. 0 ∆(A2)∆(S) D(B)= . (3) To find an appropriate description of a magnetic symme- ∆∗(S) 0 try group M, we may first start with the unitary part G. Similar to the conventional case, we can act symme- Next we need to check the condition when both the matri- try operators in G on the eigen states ψj j = 1, , d ces (2) and (3) can be diagonalized by an unitary trans- of a system. This leads to a d-dimensional| i representa-··· formation U simultaneously. There are three different tion, denoted as ∆, of the unitary group G. For the cases. If ∆(S) is complex for some S G (case c), to D S ∈ whole magnetic symmetry group M, we take ψj and keep ( ) in Eq. (2) diagonal, we require the transfor- | i U A ψj as basis where A is an anti-unitary operator in M. mation matrix to be diagonal. However, any diagonal As| showni in details in the appendix, the obtained ma- unitary tranformation matrix U cannot diagonalize D(B) trices cannot form conventional representations due to in Eq. (3). So the 2D co-representation is irreducible in the anti-unitary operators. Thus, they are dubbed “co- the case c. representations”, denoted as D below. There are two For a real representation ∆, ∆(S) = ∆∗(S) can only take the values of 1, so ∆(A2) = 1 (A2 G). Eq. main differences between co-representations and conven- ± ± ∈ tional representations. Firstly, the multiplication rules (2) is an identity matrix, and will not be changed by any are modified to (A6) (A9) for a co-representation in the unitary transformation U. Thus, we only need to find appendix. Secondly,∼ the character (the trace of a repre- a transformation matrix U to diagonalize Eq. (3). For 2 sentation matrix) can be changed by an unitary transfor- ∆(A ) = 1 (case a), one can take the unitary transformation of the basis in the co-representation theory. Con- mation squently, the character table does not have any meaning. 1 1 1 Therefore, we can not apply the conventional method to U = (4) √2 1 1 determine the reducibility of a co-representation by its − character table. Since the relation between the degener- and D(B) is transformed as acy and an irreducible co-representation still exists, the 1 ∆(S) 0 central goal of the co-representation theory is to develop a D′(B)= U − D(B)U ∗ = − method to analyze the reducibility of a co-representation 0 ∆(S) of a magnetic symmetry group. This is achieved by a B = AS AG. (5) method called the “Herring rule” [47], which will be de- ∈ scribed below. In this section, we would like to discuss Therefore, the co-representation is reducible in the case a. some simple but most useful cases and show how the 2 For ∆(A ) = 1 (case b), D(B) = i∆(S)σy for Herring rule works. − − ˆ B = AS where ~σ denote Pauli matrices. Any unitary Let us focus on the case that A = ΘR is the only gen- i~σ θ~ erator in a magnetic symmetry group M. Since TR sym- transformation matrix U can be expanded as U = e · . metry operator Θˆ commutes with any crystalline sym- Based on Eq. (A11), direct calculation gives iσj iσj iσj iσj metry operator R, the magnetic symmetry group M and D′(B)= e− ( i∆(S)σy)e− = e− e ( i∆(S)σy) its unitary subgroup G are Abelian. Consequently, the − − = i∆(S)σy = D(B) j = x, z (6) unitary subgroup G only possesses one-dimensional (1D) − irreducible representations. For a wave function ψ that and | i forms a basis for the 1D representation ∆ of the group iσy iσy D′(B)= e− ( i∆(S)σy)e G, the matrix element ψ A ψ is given by − h | | i = i∆(S)σy = D(B). (7) 2 2 − ψ A ψ = A ψ Aψ = ∆∗(A ) ψ A ψ (1) h | | i h | i h | | i Therefore, any unitary transformation U leaves the representation matrix D(B) unchanged. We conclude that 2 2 in which A G. Therefore, if ∆∗(A ) = 1, ψ A ψ the co-representation is also irreducible in the case b. ∈ 6 h | | i must equal to zero, implying that ψ and A ψ are or- In summary, according to the representation ∆ of the | i | i thogonal to each other. This argument indicates that the unitary part G, the co-representation of a magnetic point reducibility of the co-representation of M is determined symmetry group M is reducible for the case a and irre- by the representation ∆ of the unitary group G. ducible for the case b and c. Although the above discus- Now let us check the reducibility of the co- sion is based on a simple situation with only one gener- representation D directly. The matrix forms of the co- ator, a simple rule to determine the reducibility of a co- representation D are constructed as Eq. (A4) and (A5), representation exists, which was first described by Her- which can be simplified in our Abelian case. Since any ring [47]. One needs to calculate the summation of the 4 characters χ(B2) for all B AG (B2 G) and the ob- group, denoted as Mk for a magnetic space symmetry ∈ ∈ tained value determines which case the co-representation group and Gk for its unitary part. The symmetry oper- belongs to. The Herring rule can be summarized as ations in a wave vector group are different for different k, and it is more convenient to discuss the representation G in case a, of a wave vector group. For an element S t + v in a 2 | | χ(B )=  G in case b, (8) { | } −| | wave vector group at k, the representation takes the form BXAG  0 in case c. ik (t+v) ∈ ∆k( S t + v )= e · ∆(S), where ∆(S) denotes the representation{ | } for the point symmetry part. where χ is the character of the representation ∆, G is For a generic k, the wave vector group may only con- the element number of the group G. As an example,| | we tain translation symmetry operations. But at high sym- check this rule for our simple case, metry momenta, point symmetry operations and anti- χ(B2)= χ(A2S2)= χ(A2) χ2(S). (9) unitary operations can also exist. Here we are only inter- ested in the anti-unitary operation A = RΘ.ˆ We denote BXAG SXG SXG ∈ ∈ ∈ the momenta k which is invariant under the operation A Since G is a cyclic group, 1D representation can only as Kα. Kα satisfies the condition i2π n take the form of e N where N should be a factor RKα = Kα + g, (10) of G ( G = MN where N and M are integers) − | n| | | ,...,N χ2 S and = 1 . The summation S G ( ) = α , , i4π n ∈ where = 1 2 denotes the number of the invariant N ··· M n=1, ,N e has the following possibleP values. If momenta and g is a reciprocal lattice vector. These mo- ··· 2 NP > 2, B AG χ(B ) = 0 corresponds to the case c menta are dubbed “A-invariant momenta” below. The ∈ of complexP representations. If N = 1, 2, all the repre- Z2 topological invariant can be defined on these A- sentations are real and the character χ(S) can only be invariant momenta Kα in magnetic systems. 2 1 for any S G, so S G χ (S)= NM = G . Corre- For the case of TR invariant TIs, one introduces an ± ∈ ∈ 2 | | 2 ˆ spondingly, the summationP B AG χ(B )= χ(A ) G is anti-symmetric matrix wnm(Kα)= ψn,Kα Θ ψm,Kα at 2∈ | | h | | i determined by the sign of χP(A ), which just corresponds TRIM Kα. The Z2 topological invariant is defined as[48] to the case a or b. Therefore, the Herring rule (8) does det[w(Kα)] distinguish three different cases. ( 1)∆ = , (11) − pPf[w(K )] Yα α

C. Magnetic space group and Z2 topological where α = 1, 2, 3, 4. ”det” and ”Pf” are determinant invariants and Pfaffian of the anti-symmetric matrix w. It has been shown that one Z2 topological invariant can be defined Next we will illustrate our approach to determine the at four TRIM in one plane. In two dimensions, there existence of Z2 topological phases for a magnetic crys- are four TRIM, so only one Z2 invariant is possible. In tal with a given magnetic symmetry group. Our goal three dimensions, there are eight TRIM in total and as a is to construct Z2 topological invariants, in analogy to result, one can define four Z2 invariants (one strong Z2 that defined for TR invariant TIs. From the examples of invariant and three weak Z2 invariants). TR invariant TIs, Z2 topological invariants are defined For magnetic systems, the anti-unitary operator A at TRIM in the Brillouin zone (BZ). Therefore, in this plays the role of TR symmetry operator Θˆ in TR in- section, we need first to apply the Herring rule to un- variant TIs. But there are two main differences. (1) In- derstanding degeneracies in the BZ of a magnetic crystal stead of TRIM, we require at least four A-invariant mo- with a magnetic space symmetry group. menta that satisfy the condition (10), in order to define An element in a space group can be represented by a Z2 topological invariant. (2) From the Herring rule, S t + v where S is a point group operation, t is a we know that double degeneracy only occurs when the {lattice| translation} operation and v is zero or a non- states belong to the co-representation of the case b and primitive translation operation. For a magnetic space c. Only in these cases, one can define an anti-symmetric A symmetry group M(G), besides the conventional opera- matrix nm(Kα) = ψn,Kα A ψm,Kα , where ψn,Kα is h | | i | i tions, we have additional anti-unitary operators with the the eigen-state at invariant momenta Kα. By defining A K form A S t + v , where A = Θˆ R w . Here R w is √det ( α) δKα = A K , the corresponding Z2 topological in- { | } { | } {G| } P f[ ( α)] not a symmetry operation in the unitary part . For variant for TMCIs is given by a space group, all the translation operations form an ∆ Ablien subgroup, which is usually described by the 1D ( 1) = δK (12) eik t k − α representation · . Here is the wave vector that la- α=1Y, ,4 bels the 1D representation. A point symmetry opera- ··· tion usually changes one momentum k to another k′. Eq. (12) defines the Z2 topological invariant of mag- All the symmetry operations, that preserve the momen- netic systems with anti-unitary symmetry A. However, tum k, form a subgroup of the whole space group. This this definition is not convenient for the practical calcu- subgroup is known as a wave vector group or a little lation in a realistic model since a global gauge should 5

be chosen for the wave function ψn,kα . This diffi- culty can be overcome by considering| thei Wilson loop method [45, 46]. This is because bulk topological invariant can be interpreted through a simple physical pic- ture, partner switching of Wannier center flow [48]. Let’s assume the system has four A-invariant momenta Kα (α = 1a, 1b, 2a, 2b), which fall in two parallel lines K˜ 1 and K˜ (K a, b K˜ and K a, b K˜ ). For simplic- 2 1 1 ∈ 1 2 2 ∈ 2 ity, we can assume these two lines are along kz direction, which can be satisfied in all our cases by choosing appropriate coordinates. We can denote Kα = (~κα, kz) where α = 1, 2 and ~κα = (kα,x, kα,y), so ~κα is fixed for each line. Similar to the case of TR invariant TIs, we focus on doubly degenerate states or doublets. For each FIG. 1. (a) Crystal structure of C4Θ invariant tight-binding doublet, we can pick out one state ψn,kα and define I II | i ψ ψ k ψ A ψ k model. Atoms in blue and red have opposite z-directional n,kα = n, α and n,kα = n, α , where the integer| ni =1| , 2, i is for| all occupiedi | doubleti states. For magnetization. (b) The bulk Brillouin zone for our tight- ··· binding model. the fixed kx and ky, one can treat the system as a 1D system along the z direction and define a partial polarization for these doublet pairs as III. TIGHT-BINDING MODELS π i 1 i i P (~κ)= ψn,k i∂kz ψn,k dkz (i = I,II). 2π Z πh | | i A. C4Θ model Xn − (13) The partial polarization P i is directly related to the Wan- Our first model for TMCIs possesses the combined θ ψi ˆ ˆ nier function center of the occupied states n,kα by symmetry of TR Θ and four-fold rotation symmetry C4. i | i θi(~κ)=2πP (~κ). With this definition, the Wannier func- Consider a layered magnetic structure along the z di- tion centers are periodic with 2π. The Wannier function rection with a square lattice in the x-y plane. At one centers, as well as the partial polarizations, can be de- lattice site, there are four atoms with z-directional mag- rived based on the Wilson loop technique. Allowing kz netic moments and the magnetic moments between two to take discrete values, we define a U(2N) Wilson loop neighboring atoms are opposite, as shown in Fig. 1(a). for the fixed kx and ky [45], In one layer, there are two sub-layers labeled by A and B. In-plane lattice vectors are given by ~a1 = (a, 0, 0) D(~κ)= S0,1S1,2S2,3...SNz 2,Nz 1SNz 1,0 (14) and ~a2 = (0, a, 0), and the layers are stacked along − − − ~a3 = (0, 0,c). The distance between two neighboring Here, 2N 2N overlap matrices Sm,n are defined using × sub-layers is denoted as d. the periodic parts of Bloch wave-functions: First, let us analyze symmetry properties of our sys- m,n tem and show that a 2D irreducible co-representation S (~κ)= m, kz,i, ~κ n, kz,i+1, ~κ i,i+1 h | i could exist in our system. For the lattice structure shown 2πi kz,i = (15) in Fig. 1, due to the opposite alignment of magnetiza- Nza tion at the red and blue sites, both TR symmetry Θˆ ˆ where i 0, 1, ..., N 1. Then the phases of the eigen- and four-fold rotation symmetry C4 are broken. How- z ˆ ˆ ˆ values of∈ this U(2N)− matrix D(~κ) give us the Wannier ever, the combined symmetry Π = C4Θ exists for our function centers θ(~κ) of the occupied bands. system. It can be easily checked that the symmetries in ˆ ˆ ˆ ˆ3 ˆ ˆ ˆ ˆ Similar to the case of TR invariant TIs, the differ- our system are : E, C4Θ, C4 Θ, C2,m ˆ xΘ,m ˆ yΘ,m ˆ 110 I andm ˆ 1 10. Herem ˆ x,m ˆ y,m ˆ 110 andm ˆ 1 10 are mirror ence between two partial polarizations n~κα = P (~κα) − − II − reflections with the following mirror planes: (100), (010), P (~κα) can only be an integer due to A symmetry. The (110) and (1-10). Therefore, the point symmetries of our Z2 topological invariant is related to n~κα by system belong to the magnetic group C4v(C2v), where C is the magnetic point symmetry group and C is ∆= n~κ1 n~κ2 mod 2. (16) 4v 2v − its unitary part. In fact, mirror symmetries are not the Therefore, by evaluating the Wannier funtion centers essential symmetry in our system, which will be shown with the Wilson loop method, one can obtain the Z2 later by including terms that break mirror symmetries. topological number in a magnetic system with an anti- Thus, for the analysis below, we will ignore mirror sym- unitary A symmetry. Below, we will first apply our gen- metries and only consider C4(C2) group for simplicity. eral theory to two concrete models of TMCIs and then The unitary part C2 contains two 1D irreducible repre- extend the discussion to general magnetic crystal struc- sentations, A and B, the characters of which are given by tures. χA(Eˆ) = χB(Eˆ) = 1 and χA(Cˆ ) = χB(Cˆ )= 1. To 2 − 2 6 see the reducibility of the co-representation of C4(C2), representation for Cˆ4Θˆ is 2 we could calculate B GΘ χ(B ) based on the Herring ∈ P ˆ ˆ ˆ3 ˆ rule (8). In C4(C2), B GΘ can be either C4Θ or C4 Θ. ˆ ˆ 0 1 2 ∈ DB(C4Θ) = − . (17) In both cases, B = Cˆ2. Then it is easy to show that 1 0

2 χA(B )= 2 casea, This implies that it is possible to achieve double degen- BXGΘ ˆ ˆ ∈ eracies in our C4Θ model. 2 χB(B )= 2 case b. From the general co-representation theoretical analy- − BXGΘ sis, we have shown the Cˆ4Θˆ symmetry in our system ∈ could support doubly degenerate states. Next we will So for the 1D representation B (∆B (Cˆ )= 1), C (C ) show the tight-binding Hamiltonian for our system. Let 2 − 4 2 possesses a 2D irreducible co-representation which be- us consider a spinless model with px and py orbitals for longs to case b. Especially, the 2D irreducible co- each atom. The Hamiltonian is given by

H = HA + HB + HAB α,β η η α,β η η Hη = t η η cα†(~mi,n)cβ(~mj ,n)+ [t η η cα†(~mi,n)cβ(~mi+2 + ~ai,n) ~mi , ~mj ~mi , ~mi+2+~ai i=j 1,2,X3,4 , ~m,α,β,n i 1,2X, ~m,α,β,n 6 ∈{ } ∈{ } α,β η η η η η η + t η η c †(~mi+2,n)c (~mi ~ai,n)] + ηM1[ ic † (~mi,n)c (~mi,n)+ ic † (~mi,n)c (~mi,n)] ~m , ~m ~ai α β px py py px i+2 i − − − i 1,2X,3,4 , ~m,n ∈{ } α,β A B α,β A B HAB = t A B cα†(~mi,n)cβ (~mj ,n)+ [t A B cα†(~mi,n)cβ (~mi+2 + ~ai,n) ~mi , ~mj ~mi , ~mi+2+~ai i,j 1,2,3X,4 , ~m,α,β,n i 1,2X, ~m,α,β,n ∈{ } ∈{ } α,β A B α,β,int A B + t A B c †(~mi+2,n)c (~mi ~ai,n)] + t A B c †(~mi,n)c (~mj ,n + 1) (18) ~m , ~m ~ai α β ~m , ~m α β i+2 i − − i j i,j 1,2,3X,4 , ~m,α,β,n ∈{ }

where η = A, B for sub-layer A and B. M1 is the noting the index of each atom. φα denotes the wave coupling between magnetic moments and p orbitals. function of the basis with α = px,py. (~m, n) = (mx,my,n) denotes lattice sites in the layer Next we would like to apply the above general sym- n and α, β = px,py denote atomic orbitals. There are metry analysis to the Hamiltonian H(k). To do this, we four atoms in one lattice site ~m, as indicated by the need first to construct symmetry operations in the basis lower indices i and j. These four atoms locate at ~m = iηα, k for our model. The essential anti-unitary opera- 1 | i (mx + b,my), ~m2 = (mx,my + b), ~m3 = (mx b,my), tor in our system is the combined symmetry Πˆ = Cˆ4Θ,ˆ α,β − ~m4 = (mx,my b). t ~mη , ~mη is the on-site hopping ampli- which can serve as the generator of the symmetry group. − i j ˆ tude between four atoms at one lattice site (For simplic- Since our model is spinless, Θ operation is just complex α,β α,β conjugate. The behavior of basis wave-functions under ity, we denote it as t1). t η η and t η η are ~m , ~m +~ai ~m , ~m ~ai ˆ i i+2 i+2 i − Π operation is given by the nearest-neighboring hopping coefficients in sub-layer α,β ˆ η C η η ˆ η (denoted as t ). t is the nearest-neighboring Π i α, k = ( 4)iα,jβ j β, Πk (19) 3 ~mA, ~mB | i | i i j Xjη β hopping between the sub-layers A and B in the same α,β α,β Cη η layer n (denoted as t2). t A B and t A B where 4 is a block matrix defined for the sub-layer = ~m , ~m +~ai ~m , ~m ~ai i i+2 i+2 i − A, B, given by are the next-nearest-neighboring hopping coefficients be- CA tween the sub-layers A and B in the same layer n (de- C 4 0 α,β,int 4 = CB noted as t4). t A B is the nearest-neighboring inter- 0 4 ~mi , ~mj layer hopping from the sub-layer A in layer n to sub-layer where B in layer n + 1 (denoted as t5). 0 U 0 0 The Bloch Hamiltonian H(k) is given in Appendix η 0 0 U 0 0 1 C =   ,U = − . B, which is a 16 16 matrix written under the basis 4 0 0 0 U 1 0 η 1 ×i~k ~rn n U 0 0 0  i α, k = e · η,i φα(~r ~r ). Here N is the nor-   | i √N n − η,i malization factor.P The position of each atom is shown Here, the 2 2 matrix U is defined on the basis px and n ~ ~ Cη × by ~rη,i = tn + ~rη,i with tn denoting the lattice vector, py. 4 gives the rotation matrix of four atoms at one site η = A, B denoting the sub-layers and i 1, 2, 3, 4 de- in the sub-layer η. ∈ { } 7

By acting the above symmetry operations on our Bloch Hamiltonian H(k), we ﬁnd that the Hamitlonian satisﬁes the combined symmetry Π,ˆ

1 ΠˆH(kx, ky, kz)Πˆ − = H( ky, kx, kz). (20) − − According to Eq. (20), it is easy to see that the Hamilto- nian commutes with the operator Πˆ at four Πˆ invariant momenta Γ, Z, A, M, shown in Fig. 1 (b). Therefore, all the eigenstates at these Π-invariantˆ momenta can be classified by the co-representation of C4(C2) group. It is convenient to first identify which co-representation the basis iηα, K belongs to, for K =Γ,Z,A,M. This is | i FIG. 2. (a) Energy dispersion of a slab geometry config- because the coupling matrix elements of the Hamiltonian − − must be zero for two basis that belong to different co- uration with the parameter tp1 = 0.4,tp2 = 0.3,ts1 = 0.6,ts = 0.3,tp = −0.1,tp = 1,ts = 0.25,ts = 3.5, M = representations. From the general theory above, the co- 2 3 4 3 4 1 4.5,tp5 = 3.3,ts5 = 3.5 (Please see Appendix B for defini- representations of C4(C2) are classified by the eigenvalues tions of coefficients here). Red and green lines represent sur- of Cˆ2 (Eq. 17). Since [H(K), Cˆ2] = 0, H(K) and Cˆ2 face states on top and bottom surfaces. The corresponding have common eigen-states. Thus, every energy eigen- evolution of Wannier function centers is shown in (b), where state is also labeled by an eigenvalue of Cˆ2. If the Cˆ2 topological invariant ∆ = 1. eigenvalue of a state is 1, this state belongs to a doublet pair at Πˆ invariant momenta.− We would expect an eigen- state with a Cˆ eigenvalue of 1(+1) could be written ˆ 2 − At Π invariant momenta, the relations above give us ex- as a linear combination of the corresponding basis wave actly the 2D irreducible co-representation of Cˆ4Θˆ (Eq. functions. We perform a unitary transformation of basis (17)). wave-functions and find two sets of linear combinations The above symmetry analysis has indicated that topo- of basis functions: logical phases protected by Πˆ symmetry could exist in η η η η η Φ k = 1 px, k + 2 py, k 3 px, k 4 py, k our model Hamiltonian. To confirm this phase, we per- | s1, i1 | i | i − | i − | i η η η η η form numerical calculations of topological surface states,  Φ k = 1 px, k 2 py, k 3 px, k + 4 py, k  | s2, i1 | i − | i − | i | i as well as topological invariants. We carry out an energy  η η η η η  Φd ,k 1 = 1 px, k + 2 py, k + 3 px, k + 4 py, k dispersion calculation for our model Eq.(18) in a slab | 1 i | i | i | i | i η η η η η geometry along the z direction, as shown in Fig. 2(a).  Φd2,k 1 = 1 px, k 2 py, k + 3 px, k 4 py, k  | i | i − | i | i − | i By choosing appropriate parameters, we find that sur-  face states appear at the Γ point of the surface BZ in the and bulk energy gap, as shown in Fig. 2(a), where red and η η η η η Φ k = 1 py, k + 2 px, k 3 py, k 4 px, k green lines are the dispersions of surface states at top and | s1, i2 | i | i − | i − | i η η η η η bottom surfaces, respectively. For each surface, there are  Φ k = 1 py, k 2 px, k 3 py, k + 4 px, k  | s2, i2 | i − | i − | i | i two branches of surface states, which connect conduction  η η η η η  Φd ,k 2 = 1 py, k + 2 px, k + 3 py, k + 4 px, k bands to valence bands and touch at the Γ point quadrat- | 1 i | i | i | i | i η η η η η ically. The analysis of the touching point shows that the  Φd2,k 2 = 1 py, k 2 px, k + 3 py, k 4 px, k  | i | i − | i | i − | i surface wave functions belong to the 2D irreducible co-  where the indices s and d represent singlets and doublets representation of C4(C2) group. Therefore, the gapless ˆ at Π-invariantˆ momenta, as we can see from their Cˆ2 surface states are indeed protected by Π symmetry. eigenvalues, In addition, we calculate bulk topological invariants of our model to confirm the topological nature of sur- ˆ η η ˆ C2 Φs ,K δ = Φ δ face states. Since there are four Π invariant momenta 1 s1,Cˆ2K | i | i ,Z,A,M  ˆ η η Γ , according to the discussion in Sec. IIC, one  C2 Φs2,K δ = Φ ˆ K δ  s2,C2 Z2 topological invariant can be defined by Eq. (12) or  | i | i  ˆ η η by Eq. (16) based on the Wannier function centers. In  C2 Φd1,K δ = Φ ˆ K δ | i −| d1,C2 i our system, the Wannier function centers can be defined ˆ η η  C2 Φd2,K δ = Φ ˆ δ along the kz direction for any fixed kx and ky. We can  d2,C2K  | i −| i track the evolution of the Wannier function centers from  where K = Γ,Z,A,M. The indices δ = 1, 2 are related kx,y = 0 (denoted as Γ)¯ to kx,y = π (denoted as M¯ ), as to the combined symmetry operator Πˆ by shown in Fig. 2(b). Four Wannier function centers in Fig. 2(b) result from two doublet states in our model. η δ η Πˆ Φ δ = ( 1) Φ δ As expected, all Wannier function centers are degenerate d1,k d2,Πˆk | i − | i at both Γ¯ and M¯ due to Πˆ symmetry. More importantly,  η δ η  Πˆ Φ k δ = ( 1) Φ δ | d2, i − − | d1,Πˆki we find that the degenerate pairs of the Wannier func-  8

FIG. 4. (a) Layered lattice structure ofτ ˆΘˆ model. Atoms with diﬀerent colors have opposite z-directional magnetic moments. Layer A and Layer B are related by a translation of a vector c. (b) BZ ofτ ˆΘˆ model and eightτ ˆΘˆ invariant momenta.

of Fu’s model to magnetic structures. As pointed out FIG. 3. After performing a rotation operation to break mirror by Fu[28], if singlet states appear near the Fermi energy, symmetry (as is shown in (a)), surface state (b) is found in the they will weaken the stability of topological surface states following parameter: tp1 = −0.4,tp2 = −0.3,ts1 = 0.6,ts2 = 0.3,tp3 = −0.1,tp4 = 1,ts3 = 0.25,ts4 = 5, M1 = 4.5,tp5 = consisting of doublet pairs. Similar situations will occur 3.3,ts5 = 3.5. The Wannier center ﬂow plotted in (c) shows in our model. a non-trivial topological invariant.

B. τˆΘˆ model tion centers switch their partners once between Γ¯ and M¯ , indicating the Z2 number ∆ = 1 for the Eq. (16). This The above Cˆ4Θˆ protected topological phases could ex- result agrees with the existence of gapless surface states ist in magnetic structures with type-IIIa magnetic space in Fig. 2(a). symmetry groups, in which the combination of point Mirror symmetry is claimed not to be essential to pro- symmetry and TR symmetry plays an essential role. tect double degeneracies and we will discuss it below. Similar topological phases can also be found in mag- Notice that mirror-invariant plane in our model is (1-10) netic structures with type-IIIb magnetic space symme- ˆ ˆ plane, where four C4Θ invariant momenta (Γ, M, A, Z) try groups. As shown in Ref. [40, 41], Z2 topologi- locate. This makes it possible to define a mirror Chern cal phases also exist in anti-ferromagnetic models with number in our system [34]. To rule out this possibility, surface states protected by the combination of the TR we perform a rotation operation to the four atoms at one operation Θˆ and the translation operationτ ˆ. However, lattice site, as shown in Fig. 3(a). This new configu- these models contain spins and are related to the mag- ration of atoms breaks the following mirror symmetries: netic double groups. In contrast, we consider an anti- mˆ xΘ,ˆ m ˆ yΘ,ˆ m ˆ 110 andm ˆ 1 10. The only remaining sym- ferromagnetic model with spinless fermions in Ref. [42], − metries form exactly the C4(C2) magnetic group. We which will be briefly reviewed below. In this model, 2D calculate the energy dispersion in a slab geometry along ferromagnetic square lattices are layered along z direction the z direction (Fig.3 (b)), and find that gapless surface with opposite z-directional magnetic moments in neigh- states still show up. These surface states are degenerate boring layers, as shown in Fig. 4(a). So the system has at exactly Γ,¯ which implies they are only protected by an anti-ferromagnetic structure with two atoms (A and Cˆ4Θˆ symmetry. In Fig. 3(c), we plot the Wannier func- B in Fig. 4(a)) in one unit cell. Three different orbitals, tion center evolution in this new mirror-breaking model. s , px and py , are considered, and the tight-binding As we expect, a non-trivial topological invariant ∆ = 1 Hamiltonian| i | i is| giveni in Ref. [42]. appears. Due to the arrangement of magnetic moments, the Finally, we would like to comment on the connection symmetry group of this model can be generated by the between the Fu’s model with both Cˆ4 and Θ symmetry in combined symmetry ofτ ˆΘ,ˆ whereτ ˆ is a translation op- Ref. [28] and our Cˆ4Θˆ model. In Fu’s model, Cˆ4 rotation erator by a vector c = (0, 0,c) in z direction, denoted in symmetry and time reversal symmetry are preserved sep- Fig. 4(a). The corresponding unitary part of ths symme- arately, and double degeneracies of gapless surface states try group is nothing but the translation group generated also appear at Γ [28]. Based on the co-representation by the operator ˆt = (ˆτΘ)ˆ 2 = (0, 0, 2c). The 1D repre- theory and the Herring rule, we find that the essential sentation of a translation symmetry group is labelled by mπ symmetry in Fu’s model is also the combined symme- a wave vector (or a momentum) kz = Nc (m and N try Cˆ4Θ.ˆ Therefore, our Cˆ4Θˆ model is a generalization are integers), as shown in Table I. Here we have used a 9

plotted by calculating the evolution of 1D polarization TABLE I. Character table of translation group T π P (kx, kz = 2c ) from kx = 0 to kx = π. There is no 2 N−1 kz Eˆ ˆt ˆt ... ˆt gapless surface state in Fig. 5 (a) and a trivial topological 0 1 1 1 ... 1 invariant ∆ = 0 in Fig. 5 (c), so the system belongs to 2π 4π 2(N−1)π π i N i N i N a topologically trivial phase. In Fig. 5 (b) and (d), the Nc 1 e e ... e ...... system has a non-trivial topological invariant ∆ = 1 and − π 1 -1 1 ...(−1)N 1 gapless surface states appear. Notice that surface states 2c ¯ ¯ ...... are gapless and degenerate at Z but not at Γ. This is − − − 2 (N−1)π i 2(N 1)π i 4(N 1)π i 2(N 1) π consistent with our above analysis based on the Herring 1 e N e N ... e N Nc rule.

C. Other magnetic symmetry groups

Next we will generalize our theory to other magnetic groups generated by Aˆ = Θˆ Rˆ, where Rˆ can be translation, mirror or rotation operations. The case of the translation symmetry has been well discussed for the τˆΘ model in the last section. For rotation operations, Rˆ = Cˆn where n can only be taken n = 2, 3, 4, 6 due to crystallographic restriction theorem. The Herring rule can be applied to both the single and double groups to extract the degeneracy induced by the anti-unitary operation A. Furthermore, we can identify the number of Aˆ invariant momenta in the BZ. Based on these results, we can identify which types of magnetic symmetry groups are able to host Z2 topological phases. We will discuss diﬀerent cases below separately.

FIG. 5. Energy dispersion of T Θ model with a y-directional slab geometry configuration when (a) M1 = 3.1 and (b) M1 = 1. mirror symmetry 4.5. The corresponding evolution of Wannier function centers in a 3D bulk system is shown as a function of kx for (c) We will first focus on the mirror symmetry. There are M1 = 3.1, ∆=0 and (d) M1 = 4.5, ∆ = 1. always two elements E andm ˆ Θˆ for both the single and double group cases. One should note that for the double group case, (ˆmΘ)ˆ 2 =m ˆ 2Θˆ 2 = E¯E¯ = E, where E¯ for the ˆtN Eˆ periodical boundary condition = . Notice that the rotation of 2π is different from the identity E in a double τ ˆ symmetry operation ˆΘ only exists for two momenta, group. Since the unitary part has only one element E, k = 0 and π . Based on the Herring rule (Eq. 8), one z 2c the system belongs to the case a and has no non-trivial finds that topological phase. N 1 − χ(B2)= χ[(ˆtm)2] BXGΘ mX=0 2. Rotation symmetry n = 2 ∈ N kz = 0, case a =  π (21) Similar to the mirror symmetry, there are also only two 0 kz = , case c ˆ ˆ  2c elements E and C2Θ in this case, so no topological phase is possible. π  So when kz = 2c , magnetic translation group is possible to have 2D irreducible co-representations (double degeneracies) at ˆtΘˆ invariant momenta Z, U, R, T in Fig. 3. Rotation symmetry n = 3 4 (b). When kz = 0, at the other four ˆtΘˆ invariant mo- X M Y ˆ ˆ2 ˆ ˆ ˆ ˆ ˆ2 menta, Γ, , , in Fig. 4 (b), only singlet states exist. There are six elements E, C3, C3 , Θ, ΘC3, ΘC3 for the Thus, based on the co-representation theory, we arrive at n = 3 and single group case. For the unitary part, three the same conclusion as that of the previous analysis in different 1D irreducible representations Γi,i =1, 2, 3 are Ref. [42]. listed in Table. II, in which Γ1 belongs to the case a In Fig. 5 (a) and (b), we calculate energy dispersions of while Γ2,3 belong to the case c. However, systems with τˆΘˆ model in a slab geometry along y direction. In Fig. 5 such symmetries have only two invariant momenta at Γ (c) and (d), the corresponding Wannier center flows are and Z, as is shown in Fig 6(a). In order to define a 10

TABLE III. Character table of n = 4, double group 2 Rep. E C2 E¯ C2E¯ P χ(B ) class

Γ1 1111 4 a Γ2 1-11 -1 -4 b Γ3 1i-1-i 0 c Γ4 1-i-1i 0 c

FIG. 6. BZ and high symmetry points for CˆnΘˆ lattice are drawn in: (a) n=3 and n=6 (b) n=4.

Z2 topological invariant, there should be at least four invariant momenta. Therefore, no Z2 topological phase is possible in the n = 3 and single group case. Similar argument can also be applied to the n = 3 and double group case.

TABLE II. Character table of n = 3 in the single group case. πi Here ω = e 3 . 2 2 Rep. E C3 C3 P χ(B ) case FIG. 7. Examples of four topologically inequivalent phases in Γ1 111 3 a ˆ ˆ 2 a system with C6Θ symmetry are shown in (a)-(d). (a) corre- Γ2 1 ω ω 0 c 2 sponds to a trivial phase while (b)-(d) give three distinct topo- Γ3 1 ω ω 0 c logically non-trivial phases. The corresponding Fermi surfaces (blue circles) of surface states in (001) surface are shown in (e)-(h). In (i) and (j), we show another possible topological phase with an even number of gapless surface states, which is topologically equivalent to the case with an odd number of 4. Rotation symmetry n = 4 gapless surface states in (b) and (f).

The Cˆ4Θˆ symmetry has been discussed in the Cˆ4Θˆ model above for the single group case. Since there are group). Making use of Table. II, we have Γ1 belonging four momenta invariant under Cˆ4Θˆ operation in the BZ to class a, and Γ2,3 belonging to class c, so double degen- ( Fig. 6 (b)), one Z2 topological invariant can be defined eracy is possible for Γ2 and Γ3 representations. Different ˆ ˆ for spinless fermions with the Cˆ4Θˆ symmetry. from C3Θ symmetry, six momenta (Fig. 6 (a)) are invari- For the double group case, there are eight ele- ant under the Cˆ6Θˆ operation. Then we could define one ˆ ˆ ¯ ˆ ˆ ¯ ˆ3 ¯ ˆ ¯ ˆ ˆ ˆ ˆ3 ments E, ΘC4, EC2, ΘEC4 , E, ΘEC4, C2, ΘC4 . Its uni- topological invariant (Eq. 12) for each of the following ˆ ˆ tary part is the double group of Cˆ2 symmetry, which planes that contains four C6Θ invariant momenta: possesses four 1D irreducible representations, denoted as ∆1 Plane Γ Z H′ K′ : ( 1) = δΓδZ δH′ δK′ Γ1,2,3,4 in Table. III. We find that Γ2 belongs to case − − − − ∆2 b and Γ belongs to case c, so double degeneracy is Plane K′ H′ H K : ( 1) = δK′ δH′ δH δK 3,4 − − − − ∆3 also possible for Cˆ Θˆ symmetry in the double group case. Plane Γ Z H K : ( 1) = δ δZ δH δK (22) 4 − − − − Γ Four Cˆ4Θ-invariantˆ momenta( Γ, Z, M and R) allow one Z topological invariant. These three indices ∆i(i = 1, 2, 3), however, have to 2 obey the following constraint :

∆1+∆2+∆3 2 ( 1) = (δΓδZ δH δK δH′ δK′ ) 5. Rotation symmetry n = 6 − =1 (23)

The single group of Cˆ6Θˆ symmetry contains six ele- Consequently, three topologically non-trivial and in- ˆ ˆ ˆ ˆ ˆ3 ˆ2 ˆ ˆ5 ments E, ΘC6, C3, ΘC6 , C3 , ΘC6 . The unitary part in equivalent phases, as shown in Fig. 7 (b)-(d), can exist this case is the same as that of the n = 3 case (single in our Cˆ6Θˆ model, besides the topologically trivial phase 11 shown in Fig. 7 (a). Here we use red spots at Cˆ6Θˆ invari- groups are their subgroups. ant momenta Kα to represent δKα = 1 while open circles to represent δKα = 1. To see their Fermi surface struc- − K K tures, we could define πK¯ α = δ α1 δ α2 , where Kα1 and Kα2 are Cˆ6Θˆ invariant momenta that can be projected IV. DISCUSSION AND CONCLUSION to K¯ α along (001) direction. Using the same coloring convention to show the value of πK¯ , we arrive at the α Based on the Herring rule in co-representation theory corresponding Fermi surfaces and surface BZ of (001) of magnetic symmetry groups, we have developed a gen- surface plotted in Fig. 7 (e)-(h), where possible Fermi eral theory for topological magnetic crystalline insula- surfaces of surface band structures are shown in blue cir- tors. we have shown Z2 topological phases can exist in a cles. Let us take the case where (∆1, ∆2, ∆3) = (1, 0, 1) system with the Cˆ4Θorˆ τ ˆΘˆ model while Z2 Z2 topolog- as an example. In Fig. 7 (b) and (f), πΓ¯ = δΓδZ = 1. ical phases can exist in a system with the Cˆ ×Θˆ symmetry. This indicates one single Dirac-cone-like gapless surface− 6 ¯ This result extends the previous discussions [40–42] from structure at Γ. As is shown in Fig. 7 (f), Fermi energy magnetic translational group to magnetic point groups will cross surface bands an even(odd) number of times or magnetic space groups. One can easily extend our re- along the path K¯ K¯ (Γ¯ K¯ and Γ¯ K¯ ). However, we − ′ − − ′ sults to other magnetic symmetry groups with more than also found another topologically equivalent phase with one generator. To search for realistic magnetic materials, the same topological invariants and the same number of one should first identify crystal structures and magnetic Fermi level crossing, but with two Dirac-cone-like gapless ˆ ¯ ¯ structures of a system. For example, C4 symmetry ex- surface structures at K and K′, shown in Fig. 7 (i) and ists in a large group of materials with cubic symmetry, (j). It can be checked that by adding surface impurity such as face-centered cubic lattice, perovskite crystals, bands, the case in Fig. 7 (b) and (f) could be adiabat- etc. In a cubic lattice, there are different antiferromag- ically connected to the case in Fig 7 (i) and (j) without netic structures, including A-type, C-type and G-type closing the band gap. (e.g. see the Fig. 2 in Ref. [49]). It is easy to see Constrained by Eq. 23, only two topological invari- that C-type and G-type have Cˆ Θˆ symmetry combining ants are independent from each other. Different from 4 Cˆ4 rotation along the z direction and TR operation. A the strong and weak indices for TR-invariant TIs, all large variety of materials with perovskite structures, such three topologically non-trivial phases share the same sta- as SrMnO3[49], PbFeO2F[50], etc, possess these symme- bility. Therefore, without loss of generality, we can tries. One needs to further search for semiconducting Z , choose a 2 topological invariant pair (∆1 ∆2) to clas- materials with a narrow gap in these systems in order to sify Z Z topological phases in a Cˆ Θˆ model. When 2 × 2 6 realize topologically non-trivial phases, which is beyond (∆1, ∆2) = (0, 0), the system is topologically trivial. the scope of this paper. When (∆1, ∆2)=(0, 1), (1, 0), (1, 1), the system is topologically non-trivial. For the double group, there are also six elements: Cˆ6Θ,ˆ ˆ2 ¯ ˆ3 ¯ ˆ ˆ4 ˆ5 ˆ C6 E, C6 EΘ, C6 , C6 Θ, E and the character table for the V. ACKNOWLEDGMENTS unitary part is shown in Table. IV. Similar to the single group case, double degeneracy exists for Γ2,3 representations based on Table. IV. One Z Z topological We would like to thank C. Fang, X.L. Qi and S.C. 2 × 2 invariant pair (∆1, ∆2) could be defined among three Z2 Zhang for useful discussions. topological invariants ∆1,2,3 to classify the topological phases protected by Cˆ6Θˆ double group symmetry.

Appendix A: Review of co-representation theory for TABLE IV. Character table of n = 6, double group. Here magnetic symmetry groups 2πi ω = e 3 . 4 ¯ 2 2 Rep. E C6 EC6 P χ(B ) class Now we consider the magnetic symmetry group M =

Γ1 111 3 a G + AG and assume the wave functions ψj (j = 2 | i Γ2 1 ω ω 0 c 1, , d) form a set of basis for the unitary symmetry 2 ··· Γ3 1 ω ω 0 c group G. The corresponding representation is defined as S ψj = ∆ij (S) ψi , which satisfies ∆(SW ) = | i i | i ∆(S)∆(W ) (S,P W G). Let’s define φj = A ψj , and Up to now, we have shown that besides Kramers’ de- ∈ | i | i generacy due to TR symmetry Θ,ˆ systems with Cˆ4Θˆ or 1 Cˆ Θˆ symmetry operations are able to support topolog- S φj = SA ψj = A ∆ij (A− SA) ψi 6 | i | i | i ical phases for both spinless and spinful cases. This re- Xi 1 sult can be applied to more complicated magnetic point = ∆∗ (A− SA) φi (A1) ij | i groups and magnetic space groups when magnetic cyclic Xi 12

1 where we have used A− SA G. For any B AG, one S ϕi′ = j SUji ϕj = kj UjiDkj (S) ϕk = ∈ ∈ | i 1 | i | i has U − DkjP(S)Uji ϕ′ , andP similarly B ϕ′ = kjl lk | li | ii 1 1 P BUji ϕj = U ∗ Dkj (B) ϕk = B ψj = A(A− B) ψj = A ∆ij (A− B) ψi i jk ji | i | i | i |1 i | i Xi P U − Dkj (B)U ∗ ϕ′ , so P jkl lk ji| li 1 P = ∆ij∗ (A− B) φi (A2) | i 1 Xi D′(S)= U − D(S)U (A10) 1 B φj = BA ψj = ∆ij (BA) ψi (A3) D′(B)= U − D(B)U ∗. (A11) | i | i | i Xi From the above equaitons, it is easy to know that ϕ ϕ ψ ϕ φ Therefore , deﬁned as j = j and j+d = j , T r(D (B)) = T r(D(B)), so the character, as well as the | i | i | i M| i | i ′ form the basis for the symmetry group , and the cor- character table,6 has no meaning for co-representation. responding “representation” D is deﬁned as Although the character and character table can not ∆(S) 0 work for co-representations, the general relation between D(S)= 1 (A4) 0 ∆∗(A− SA) degeneracy and an irreducible representation still exist. By construction, our co-representation D has at least the 0 ∆(BA) D(B)= (A5) dimension two. Therefore, one need to study the irre- ∆ (A 1B) 0 ∗ − ducibility of the co-representation D. It turns out that there is a simple rule to determine the reducibility of the where S ϕj = i Dij (S) ϕi and B ϕj = | i | i¯ | i1 co-representation D, known as the Herring rule[47]. The i Dij (B) ϕi . We usuallyP denote ∆(S) = ∆∗(A− SA) | i Herring rule can be summarized as whichP also forms a representation for unitary group G. Although D is consisted of two sets of representations, it is not the conventional representation by itself, which G in case a, 2 | | can easily be seen by multiplication rules. χ(B )=  G in case b, (A12) −| | BXAG  0 in case c. 0 ∆(BSA) ∈ D(BS)= ∆ (A 1BS) 0  ∗ − where χ is the character of the representation ∆, G 1 | | 0 ∆(BA)∆(A− SA) is the element number of the group G. The case a is = 1 ∆∗(A− B)∆∗(S) 0 reducible while the cases b and c are irreducible.

0 ∆(BA) ∆∗(S) 0 = 1 1 ∆∗(A− B) 0 0 ∆(A− SA)

= D(B)D∗(S) (A6) Appendix B: Bloch Hamiltonian of Cˆ4Θˆ -invariant topological insulator Therefore, D does not obey the multiplication rule of the conventional representations, and it is usually called co- M In this section, we will present the detailed form of our representation of magnetic symmetry group [43, 44]. ˆ ˆ Other multiplication rules can be obtained as our C4Θ-invariant tight-binding model in the momenm- tum space. This Bloch Hamiltonian is given by a 2 2 × D(S)D(W )= D(SW ) (A7) block matrix. D(S)D(B)= D(SB) (A8) HA HAB D(B)D∗(C)= D(BC) (A9) H = (B1) HAB† HB It is important to note that for a unitary transformation U, ϕ′ = Uji ϕj , one has Each block HA,B is an 8 8 matrix, as is shown below. | ii i | i × P 13

ib(ky −kx) ib(ky −kx) 0 iM c e (tp +ts ) c e (tp ts ) − 1 0 1 1 0 1− 1 ib(ky −kx) ib(ky −kx) iM 0 c e (tp ts ) c e (tp +ts ) 11 1 0 1− 1 0 1 1 HA =  ib(kx −ky ) ib(kx −ky )  c e (tp +ts ) c e (tp ts ) 0 iM 0 1 1 0 1− 1 1 ib(kx −ky ) ibi(kx −ky ) c e (tp ts ) c e (tp +ts ) iM 0  0 1− 1 0 1 1 − 1  ib(kx −ky ) ib(kx −ky ) 0 iM c e (tp +ts ) c e (tp ts ) − 1 0 1 1 0 1− 1 ib(kx −ky ) ib(kx−ky ) iM 0 c e (tp ts ) c e (tp +ts ) 22 1 0 1− 1 0 1 1 HA =  ib(ky −kx) ib(ky −kx)  c e (tp +ts ) c e (tp ts ) 0 iM 0 1 1 0 1− 1 1 ib(ky −kx) ib(ky −kx) c e (tp ts ) c e (tp +ts ) iM 0  0 1− 1 0 1 1 − 1  −i2bkx i2¯bkx ib(−kx−ky ) ib(−kx −ky ) e ts +e ts 0 c e (tp +ts ) c e ( tp +ts ) 1 3 0 1 1 0 − 1 1 −i2bkx i2¯bkx ib(−kx−ky ) ib(−kx −ky ) 0 e tp +e tp c e ( tp +ts ) c e (tp +ts ) 12  1 3 0 − 1 1 0 1 1  HA = ib(−ky −kx) ib(−ky −kx) −i2bky i2¯bky c e (tp +ts ) c e ( tp +ts ) e tp +e tp 0 0 1 1 0 − 1 1 1 3 ib(−ky −kx) ib(−ky −kx) −i2bky i2¯bky  c e ( tp +ts ) c e (tp +ts ) 0 e ts +e ts   0 − 1 1 0 1 1 1 3  21 12 HA = (HA )† HB = HA (hopping coeﬃcients (including M1) with an opposite sign) −ibkz i¯bkz e tp2+e tp5 0 a1 a¯1 −ibkz i¯bkz 11  0 e tp2+e tp5 a¯1 a1  HAB= −ibkz i¯bkz a2 a¯2 e tp2+e tp5 0 − ¯  a¯ a 0 e ibkz t +eibkz t   2 2 p2 p5  H22 = H11 (e e ,e e ) AB AB 1 → 2 2 → 1 a3 0 a5 a¯5 12  0 a4 a¯5 a5  HAB= a5 a¯5 a¯4 0  a a a   ¯5 5 0 ¯3  21 12 H = H (kx kx, ky ky) (B2) AB AB →− →−

1 For each hopping amplitude, we have used lower in- c4 = 3 andc ¯i = 1 ci. These ci andc ¯i are re- dices s and p to label σ and π bond contribution. For lated to directional cosines− of atom configurations. We ib( kx+ky kz ) ib(kx ky kz ) example, ts1 is the σ contribution for t1, while tp1 is the also define e1 = e − − , e2 = e − − , ib( kx ky kz ) ikz π contribution for t1. We define b = 0.1, ¯b = 1 b e3 = e − − − ,¯ei = eie and the notations below and 2¯b = 1 2b. So √2b is the distance between− for simplicity. All the a parameters are listed as follows. nearest neighboring− atoms in one unit cell. We define 1 c0 = 2 , c1 = 0.987952, c2 = 0.984615, c3 = 0.952941,

a1 = e1(¯c4tp2 + c4ts2)+¯e1(c1tp5 +¯c1ts5)

a¯ = e (c tp c ts )+¯e (¯c tp c¯ ts ) 1 1 4 2 − 4 2 1 1 5 − 1 5 a2 = e2(¯c4tp2 + c4ts2)+¯e2(c1tp5 +¯c1ts5)

a¯ = e (c tp c ts )+¯e (¯c tp c¯ ts ) 2 2 4 2 − 4 2 2 1 5 − 1 5 i( 2bkx bkz ) ikx ikz a3 = e − − ((2btp2 + 2¯bts2)+ e (¯c2tp4 + c2ts4)+ e (c3tp5 +¯c3ts5))

i( 2bky bkz ) iky ikz a¯3 = e − − ((2btp2 + 2¯bts2)+ e (¯c2tp4 + c2ts4)+ e (c3tp5 +¯c3ts5))

i( 2bkx bkz ) i(2¯bkx bkz) i(¯bkz 2bkx) a4 = e − − tp2 + e − tp4 + e − tp5

i( 2bky bkz ) i(2¯bky bkz) i(¯bkz 2bky ) a¯4 = e − − tp2 + e − tp4 + e − tp5

a5 = e3(¯c4tp2 + c4ts2)+¯e3(c1tp5 +¯c1ts5)

a¯ = e (c ts c tp )+¯e (¯c ts c¯ tp ) (B3) 5 3 4 2 − 4 2 3 1 5 − 1 5

[1] X.-L. Qi and S.-C. Zhang, Rev. Mod. Phys. 83, 1057 [2] M. Z. Hasan and C. L. Kane, Rev. Mod. Phys. 82, 3045 (2011). (2010). 14

[3] J. E. Moore, Nature 464, 194 (2010). [27] C. Xu and J. Moore, Phys. Rev. B 73, 045322 (2006). [4] X. L. Qi and S. C. Zhang, Phys. Today 63, 33 (2010). [28] L. Fu, Phys. Rev. Lett. 106, 106802 (2011). [5] K. v. Klitzing, G. Dorda, and M. Pepper, Phys. Rev. [29] R.-J. Slager, A. Mesaros, V. Juriˇcić, and J. Zaanen, Na- Lett. 45, 494 (1980). ture Physics 9, 98 (2012). [6] C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 226801 [30] C. Fang, M. J. Gilbert, and B. A. Bernevig, Physical (2005). Review B 86, 115112 (2012). [7] C. L. Kane and E. J. Mele, Phys. Rev. Lett. 95, 146802 [31] X.-L. Qi, T. L. Hughes, S. Raghu, and S.-C. Zhang, Phys. (2005). Rev. Lett. 102, 187001 (2009). [8] B. A. Bernevig and S. C. Zhang, Phys. Rev. Lett. 96, [32] R. M. Lutchyn, J. D. Sau, and S. Das Sarma, Phys. Rev. 106802 (2006). Lett. 105, 077001 (2010). [9] B. A. Bernevig, T. L. Hughes, and S. C. Zhang, Science [33] L. Fu and C. L. Kane, Phys. Rev. Lett. 100, 096407 314, 1757 (2006). (2008). [10] L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, [34] T. H. Hsieh, H. Lin, J. Liu, W. Duan, A. Bansil, and 106803 (2007). L. Fu, Nature Communications 3, 982 (2012). [11] L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007). [35] P. Dziawa, B. Kowalski, K. Dybko, R. Buczko, A. Szczer- [12] J. E. Moore and L. Balents, Phys. Rev. B 75, 121306 bakow, M. Szot, E.Lusakowska, T. Balasubramanian, (2007). B. M. Wojek, M. Berntsen, et al., Nature Materials 11, [13] M. König, S. Wiedmann, C. Brüne, A. Roth, H. Buh- 1023 (2012). mann, L. Molenkamp, X.-L. Qi, and S.-C. Zhang, Science [36] Y. Tanaka, Z. Ren, T. Sato, K. Nakayama, S. Souma, 318, 766 (2007). T. Takahashi, K. Segawa, and Y. Ando, Nature Physics [14] D. Hsieh, D. Qian, L. Wray, Y. Xia, Y. S. Hor, R. J. 8, 800 (2012). Cava, and M. Z. Hasan, Nature 452, 970 (2008). [37] S.-Y. Xu, C. Liu, N. Alidoust, M. Neupane, D. Qian, [15] H. Zhang, C.-X. Liu, X.-L. Qi, X. Dai, Z. Fang, and S.-C. I. Belopolski, J. Denlinger, Y. Wang, H. Lin, L. Wray, Zhang, Nature Phys. 5, 438 (2009). et al., Nature communications 3, 1192 (2012). [16] Y. Xia, L. Wray, D. Qian, D. Hsieh, A. Pal, H. Lin, [38] M. S. Dresselhaus, G. Dresselhaus, and A. Jorio, Group A. Bansil, D. Grauer, Y. Hor, R. Cava, et al., Nature Theory: Application to the Physics of Condensed Matter Phys. 5, 398 (2009). (Springer-Verlag, Berlin Heidelberg, 2008). [17] Y. L. Chen, J. G. Analytis, J. H. Chu, Z. K. Liu, S.-K. [39] Y. L. Chen, J.-H. Chu, J. G. Analytis, Z. K. Liu, Mo, X. L. Qi, H. J. Zhang, D. H. Lu, X. Dai, Z. Fang, K. Igarashi, H.-H. Kuo, X. L. Qi, S. K. Mo, R. G. Moore, et al., Science 325, 178 (2009). D. H. Lu, et al., Science 329, 659 (2010). [18] B. Yan and S.-C. Zhang, Reports on Progress in Physics [40] R. S. Mong, A. M. Essin, and J. E. Moore, Physical Re- 75, 96501 (2012). view B 81, 245209 (2010). [19] D. Hsieh, Y. Xia, L. Wray, D. Qian, A. Pal, J. H. Dil, [41] C. Fang, M. J. Gilbert, and B. A. Bernevig, arXiv J. Osterwalder, F. Meier, G. Bihlmayer, C. L. Kane, preprint arXiv:1304.6081 (2013). et al., Science 323, 919 (2009). [42] C.-X. Liu, arXiv preprint arXiv:1304.6455 (2013). [20] P. Roushan, J. Seo, C. V. Parker, Y. S. Hor, D. Hsieh, [43] C. Bradley and B. Davies, Reviews of Modern Physics D. Qian, A. Richardella, M. Z. Hasan, R. J. Cava, and 40, 359 (1968). A. Yazdani, Nature 460, 1106 (2009). [44] E. Wigner, Group theory: and its application to the quan- [21] T. Zhang, P. Cheng, X. Chen, J.-F. Jia, X. Ma, K. He, tum mechanics of atomic spectra, vol. 5 (Academic Press, L. Wang, H. Zhang, X. Dai, Z. Fang, et al., Phys. Rev. 1959). Lett. 103, 266803 (2009). [45] R. Yu, X. L. Qi, A. Bernevig, Z. Fang, and X. Dai, Phys- [22] Z. Alpichshev, J. G. Analytis, J. H. Chu, I. R. Fisher, ical Review B 84, 075119 (2011). Y. L. Chen, Z. X. Shen, A. Fang, and A. Kapitulnik, [46] R. D. King-Smith and D. Vanderbilt, Phys. Rev. B 47, Phys. Rev. Lett. 104, 016401 (2010). 1651 (1993). [23] C. Brüne, C. X. Liu, E. G. Novik, E. M. Hankiewicz, [47] C. Herring, Phys. Rev. 52, 361 (1937), URL H. Buhmann, Y. L. Chen, X. L. Qi, Z. X. Shen, S. C. http://link.aps.org/doi/10.1103/PhysRev.52.361. Zhang, and L. W. Molenkamp, Phys. Rev. Lett. 106, [48] L. Fu and C. L. Kane, Phys. Rev. B 74, 195312 (2006). 126803 (2011). [49] R. Sønden˚a, P. Ravindran, S. Stølen, T. Grande, and [24] D.-X. Qu, Y. Hor, J. Xiong, R. Cava, and N. Ong, Science M. Hanfland, Physical Review B 74, 144102 (2006). 329, 821 (2010). [50] T. Katsumata, A. Takase, M. Yoshida, Y. Inaguma, J. E. [25] J. G. Analytis, R. D. McDonald, S. C. Riggs, J.-H. Chu, Greedan, J. Barbier, L. Cranswick, and M. Bieringer, G. Boebinger, and I. R. Fisher, Nature Physics 6, 960 in MRS Proceedings (Cambridge Univ Press, 2006), vol. (2010). 988. [26] C. Wu, B. A. Bernevig, and S. C. Zhang, Phys. Rev. Lett. 96, 106401 (2006).