Spin-orbit physics of j=1/2 Mott insulators on the triangular lattice

Michael Becker,1 Maria Hermanns,1 Bela Bauer,2 Markus Garst,1 and Simon Trebst1 1Institute for Theoretical Physics, Cologne University, Zulpicher¨ Straße 77, 50937 Cologne, Germany 2Station Q, Microsoft Research, Santa Barbara, CA 93106-6105, USA (Dated: April 16, 2015) The physics of spin-orbital entanglement in effective j = 1/2 Mott insulators, which have been experi- mentally observed for various 5d transition metal oxides, has sparked an interest in Heisenberg-Kitaev (HK) models thought to capture their essential microscopic interactions. Here we argue that the recently synthesized Ba3IrTi2O9 is a prime candidate for a microscopic realization of the triangular HK model – a conceptually interesting model for its interplay of geometric and exchange frustration. We establish that an infinitesimal Ki- taev exchange destabilizes the 120◦ order of the quantum Heisenberg model. This results in the formation of an extended Z2-vortex crystal phase in the parameter regime most likely relevant to the real material, which can be experimentally identified with spherical neutron polarimetry. Moreover, using a combination of analytical and numerical techniques we map out the entire phase diagram of the model, which further includes various ordered phases as well as an extended nematic phase around the antiferromagnetic Kitaev point.

I. INTRODUCTION yond the hexagonal lattice, triggered mainly by the synthe- sis of novel Iridate compounds, which includes e.g. the sis- 16 17 ter compounds β-Li2IrO3 and γ-Li2IrO3 that form three- The physics of transition metal oxides with partially filled dimensional Ir lattice structures. In this manuscript, we turn to 5d shells is governed by a largely accidental balance of elec- 18 the recently synthesized Iridate Ba3IrTi2O9 and argue that tronic correlations, spin-orbit entanglement, and crystal field it realizes a Heisenberg-Kitaev (HK) model on a triangular effects, with all three components coming up roughly equal lattice36. This model is of deep conceptual interest as it ex- in strength. With different materials exhibiting slight tilts to- hibits a subtle interplay of the two elementary sources of frus- wards one of the three effects a remarkably broad variety of tration – geometric frustration arising from its non-bipartite quantum states has recently been suggested, which includes lattice structure as well as exchange frustration arising from exotic states such as Weyl semi-metals, axion insulators, or 1 the Kitaev couplings. The ground states of the classical HK topological Mott insulators . A particularly intriguing sce- model on the triangular lattice have been already addressed by nario is the formation of Mott insulators in which the lo- Rousochatzakis et al.20. With the help of Monte Carlo simula- cal moments are spin-orbit entangled Kramers doublets. An tions these authors demonstrated that a small but finite Kitaev example are the j = 1 Mott insulators observed for vari- 2 exchange in addition to an antiferromagnetic Heisenberg in- 2–4 ous Iridates . The Iridium valence in the latter typically is teraction stabilizes a -vortex crystal. The -vortices can 4+ 5 Z2 Z2 Ir corresponding to a 5d electronic configuration. With be viewed as defects of the SO(3) order parameter associated the crystal field of the octahedral IrO oxygen cage splitting 6 with the 120◦ ordering of the antiferromagnetic Heisenberg off the two eg levels, this puts 5 electrons with an effective model. It is important to note that this physics plays out near 1 s = 2 magnetic moment into the t2g orbitals, which entan- the Heisenberg limit of the HK model – the relevant micro- gled by strong spin-orbit coupling leaves the system with a scopic parameter regime for all Iridates synthesized so far. As fully filled j = 3 band and a half-filled j = 1 band2,3,5. The 2 2 such Ba3IrTi2O9 is a prime candidate to observe this exotic smaller bandwidth of the latter then allows for the opening of phase. a Mott gap even for the relatively moderate electronic corre- After a discussion of the material aspect of Ba IrTi O and 1 3 2 9 lations of the 5d compounds. Interest in such j = 2 Mott a motivation of the HK model in Sec.II, we examine in insulators has been sparked by the theoretical observation6–8 Sec. III the formation of a Z2-vortex crystal from the analyti- that the microscopic interaction between their spin-orbit en- cal perspective of an expanded Luttinger-Tisza approximation tangled local moments not only includes an isotropic Heisen- and quantitatively describe its experimental signatures in po- berg exchange but also highly anisotropic interactions whose larized neutron scattering experiments. In Sec.IV we address easy axis depends on the spatial orientation of the exchange the full phase diagram of the HK model and discuss the var- 9 arXiv:1409.6972v2 [cond-mat.str-el] 15 Apr 2015 path tracing back to the orbital contribution of the moments . ious phases with the help of analytical as well as numerical In a hexagonal lattice geometry, as it is found for the lay- methods. Finally, in Sec.V we close with a summary. ered Na2IrO3 and α-Li2IrO3 compounds, these anisotropic interactions provide an implementation of the celebrated Ki- 10 taev model known for its spin liquid ground states. A trove II. MATERIAL PHYSICS OF Ba3IrTi2O9 of experimental data11, ab initio calculations12, and model 13 simulations for these hexagonal systems has spurred an on- 4+ Ba3IrTi2O9 forms layers of Ir ions in a triangular geome- going discourse illuminating the actual spin-orbital ordering try, which are separated from each other by two layers of Ti4+ mechanism in these materials. ions. An important characteristic of the Ir layer geometry il- Much recent activity14,15 has been targeted towards the lustrated in Figs.1 a) and b) is that it exhibits the two neces- physics of j = 1/2 Mott insulators for lattice geometries be- sary ingredients for Kitaev-type exchange couplings. First, 2

a) crystal structure b) single Iridium layer c) Ir-O-O-Ir exchange path

z y Ir O zˆ yˆ x zˆ yˆ zˆ xˆ yˆ Ti Ba xˆ view along (111) xˆ view along (311)

Figure 1. (Color online) a) Crystal structure of Ba3IrTi2O9. b) View of single Iridium layers from two different perspectives. Within the plane the x, y, and z exchange paths are indicated by the grey planes. The planes labeled by x (y, z) are normal to the coordinate axis xˆ (yˆ, zˆ). c) The exchange between the Iridium moments (blue) is mediated by two coplanar exchange paths. every pair of Iridium ions is coupled via two separate ex- ground state of the antiferromagnetic Heisenberg Hamilto- change paths as indicated in Fig.1 c) leading to a destruc- nian on the triangular lattice, which corresponds to couplings tive interference and subsequent suppression of the isotropic JH > 0 and JK = 0 for Hamiltonian (1), is characterized by 6–8 21 Heisenberg exchange . In comparison to the tricoordinated a 120◦ ordering of spins . At the classical level this order- ˆ Iridates (Na,Li)2IrO3, which exhibit Ir-O-Ir exchange paths, ing is captured by a spin orientation Si = SΩ(ri) with the ˆ the triangular Ba3IrTi2O9 exhibits somewhat longer Ir-O-O-Ir unit vector Ω120◦ (r) = e1 cos (Q r) + e2 sin (Q r) where exchange paths resulting in an overall lessening of the mag- the commensurate wavevector Q connects· the center· with a netic exchange strength. Second, the three principal bond di- 4π corner of the Brillouin zone, Q = 3 (1, 0). The orthonormal rections of the triangular lattice structure cut through three frame ei with i = 1, 2, 3 and e3 = e1 e2 constitutes an different edges of the IrO6 oxygen cages resulting in a distinct SO(3) order parameter. The energy per site× for this classical 6–8 locking of the exchange easy axis along the three directions state is given by as illustrated Fig.1 a) and ultimately giving rise to the three components of the Kitaev exchange. Note that the Ir layer is normal to the (111) direction, hence the three directions are 2 1  ε ◦ = S 3J + J . (2) all equivalent. The description of the microscopic physics is 120 − 2 H K thus given in terms of a Heisenberg-Kitaev (HK) Hamiltonian

X X γ γ 22 = J S S + J S S , (1) Crucially, the 120◦ ordering possesses Z2 vortices as HHK H i · j K i j ij γ ij topologically stable point defects, which can be understood h i kh i by considering the first homotopy group of its order parame- where Si is a spin-operator located on site i of the trian- ter Π1(SO(3)) = Z2. T gular lattice spanned by the lattice vectors ax = (1, 0) , T ay = ( 1/2, √3/2) , and az = ax ay, see Fig.2 a). Here and− in the following, we measure− lengths− in units of the lattice constant a. The first term is the standard Heisenberg ◦ coupling, JH , that describes an SU(2) invariant interaction be- A. Kitaev interaction destabilizes 120 ordering tween the spin-orbit entangled j = 1/2 moments on nearest- neighbor lattice sites. The Kitaev interaction, JK , on the other hand, explicitly breaks spin-rotation invariance and acts only For any finite JK the 120◦ state becomes immediately un- between single components, Sγ , of adjacent spins. The pre- stable with respect to fluctuations, which we demonstrate in cise component depends on the link between the lattice sites, the following. We parametrize the fluctuations with the help T see Fig.2 a); for our particular choice here, the γ-components of two real fields π(r) = (π1(r), π2(r)) , of spins interact via JK if sites are connected by a lattice vec- tor aγ with γ = x, y, z. q 2 Ωˆ(r) = Ωˆ ◦ (r) 1 (π(r)) (3) 120 − r e Q r e Q r r e ◦ + π1( )( 1 sin ( ) + 2 cos ( )) + π2( ) 3, III. 120 ORDER AND Z2-VORTEX CRYSTAL − · ·

We will start our discussion of the ground states of Hamilto- so that Ωˆ 2(r) = 1 is mantained. Plugging this Ansatz in the nian (1) by first elucidating the magnetic structure around the Hamiltonian and expanding up to second order in the fluctua- (2) antiferromagnetic Heisenberg point, where an extended Z2 tion fields one obtains for the energy = Nε120◦ + with vortex crystal phase is found in agreement with Ref.20. The N denoting the number of lattice sites.E The fluctuationE part 3

a) b) It becomes maximally negative for a wavevector ay x x JK 4 X γ 2 Si Sj kinst = aγ sin(Q aγ )(e ) (7) az ax y y J 3 · 3 Si Sj Q H γ=x,y,z z z Si Sj T JK  1 h y 2 z 2 x 2 i z 2 y 2 = (e3) + (e3) 2(e3 ) ) , (e3) (e3) , JH √3 − − Figure 2. (Color online) a) The triangular lattice with the three lattice vectors aγ . Solid, dashed and dotted bonds carry the three distinct that can be expressed in terms of the normal e3. In the spe- Kitaev interactions, see text. b) First Brillouin zone of the triangular cial case where the spins of the 120◦ ordering are confined lattice. The position and size of the coloured dots indicate the posi- within the x-y plane and e3 = ˆz, this wavevector is just given T tion and weight of Bragg peaks, respectively, expected in the static by kinst = JK /JH (1/√3, 1) . So it is the tilting Goldstone spin structure factor for the vortex crystal. Each color corresponds Z2 modes that trigger the instability of the 120◦ antiferromag- to a different spin-component as listed in panel a). netic ordering in the presence of a finite Kitaev interaction JK . reads

2 B. Incommensurate antiferromagnet: Z2-vortex crystal (2) X 2 JH S X = ε ◦ (π(r )) (π π 2π π ) E − 120 i − 2 1i 1j − 2i 2j i ij Indeed, allowing for a slowly spatially varying orthogonal h i X h  frame e (r) one finds in the limit J J the effective 2 γ γ γ γ Qr Qr i K H + JK S e3 e3 π2iπ2j + e1 e1 cos( i) cos( j) energy functional = R d2r with| |  γ ij E L kh i 2 γ γ γ γ  3JH S X  2  + + e e sin(Qr ) sin(Qr ) e e sin(Q(r + r )) π π = e−(r) 2iqK aγ e (r), 2 2 i j − 1 2 i j 1i 1j L 4 γ −∇ − · ∇ γ h i i γ=x,y,z + eγ sin(Qr ) + eγ cos(Qr ) eγ π π + (i j) , (8) − 1 i 2 i 3 1i 2j ↔ (4) where e± = (e ie )/√2. The Kitaev interaction induces a 1 ± 2 coupling qK = 2JK /(√3JH ) to constant gauge fields given with the abbreviation πai = πa(ri) for a = 1, 2. The fluctu- by the triangular lattice vectors aγ , that can be identified as ation eigenmodes are determined with the help of the Fourier 20 1 P ikri Lifshitz invariants as previously pointed out in Ref. . The transform πa(ri) = √ k 1.BZ e πa(k). In the absence N ∈ magnetization can thus minimize its energy by allowing for of the Kitaev interaction, JK = 0, one obtains a spatial modulation of the SO(3) order parameter on large length scales proportional to 1/q J /J . 2 h K H K (2) JH S X ∝ = (1 cos(kaγ ))π1∗(k)π1(k) JK =0 2 E k∈1.BZ − γ=x,y,z 1. Luttinger-Tisza approximation i + (1 + 2 cos(kaγ ))π2∗(k)π2(k) (5) The character of this modulated classical ground state can be obtained by minimizing the Hamiltonian treating the or- with πa∗(k) = πa( k). Whereas the π1 mode becomes soft − ˆ 2 at the center of the Brillouin zone, i.e., at k = 0, the en- thonormal constraint, ei ej = δij, or equivalently, Ω = 1, · 24 ergy of the π2 mode vanishes at its edge, e.g. for momenta within an improved Luttinger-Tisza approximation . The lat- k = Q. The zero modes π (k = 0) and π ( Q) thus ter is a good approximation for large length scales qK r 1, ± 1 2 ± | |  identify three Goldstone modes that correspond to a long- or, alternatively, for small momenta q qK . Note that this | |  wavelength rotation and tilting of the local orthogonal frame, latter limit does not commute with JK 0, and, as a conse- → respectively. In particular, the energy dispersion of the tilting quence, does not smoothly connect with the Heisenberg point. tilt 2 P We start with the functional mode εk JK =0 = JH S γ=1,2,3(1+2 cos(k aγ )) close to | tilt · 2 3 2 momentum Q possesses the form εQ+k JK =0 JH S 8 k . = (9) | ≈ E Adding a finite Kitaev coupling JK immediately results in X X X J S2 Ωˆ Ωˆ + J S2 Ωˆ γ Ωˆ γ λ (Ωˆ 2 1). a negative energy eigenvalue and, therefore, destabilises the K i · j K i j − i i − ij γ ij i 120◦ ground state. We can still diagonalize for the eigenen- h i kh i ergies perturbatively in J . In lowest order and in the long- K The unit length of the vector Ωˆ is locally imposed with the wavelength limit the zero modes do not hybridize, and we ob- i help of the Lagrange multipliers λ . Upon spatial Fourier tain for the tilting mode a dispersion relation that is given in i transformation, Ωˆ(r) = P eiqrΩˆ , the functional takes the the long-wavelength limit, k Q , by q q | |  | | form X X 3 X γ ˆ α αβ ˆ β ˆ α ˆ α εtilt J S2 k2 2J S2 k a sin(Q a )(e )2. /N = Ω q (q)Ωq + λ q pΩpΩq λ0, Q+k ≈ H 4 − K · γ · γ 3 E − J − − − γ=x,y,z q q,p (6) (10) 4

αβ where λ0 = λq q=0. The matrix J possesses only diagonal tmin(2aη aγ ) with η = γ. The resulting Bragg peaks in entries with | the− static structure− factor6 are visualized in Fig.2 b), which   nicely agrees with previous numerical findings for the clas- αα 2 20 (q) = JH S cos(ax q) + cos(ay q) + cos(az q) sical model . The relative weight of secondary and primary J · · · Bragg peaks are predicted to be 1/42 = 1/16 within the above + J S2 cos(a q) (11) K α · approximation. We find that the corresponding energy is in- dependent of the choice of origin r = (x , y )T as well as and J αβ(q) = 0 for α = β. At the Heisenberg point 0 0 0 6 the phase φ. JK = 0, the diagonal components of the matrix are min- ˆ imal for momenta at the corner of the Brillouin zone, e.g. In the Luttinger-Tisza approximation the length of the Ω vector is compromised to differ from unity, P3 (Ωˆ α(r))2 = q = Q, thus leading to 120◦ ordering. A finite J , however, α=1 t K ˆ 6 favours in general incommensurate order with wave vectors 1. Whereas the length Ωt(r) varies in space it nevertheless αα | | ˆ away from Q as J (q) become minimal for momenta of the remains always finite so that the orientation of Ωt(r) is al- (1) ways well defined. Note that in the limit J 0 the dis- form qα = Q t aα with t R. On the other hand, Fourier K − ∈ 2 → components, Ωˆ α , of the spin with such incommensurate tance tmin 0 and εLT(0) = S 3JH /2 recovers the exact (1) → − qα ground state energy whereas the state itself, Ωˆ α (r), does not wave vectors induce finite Fourier components, λ (1) with t=0 2qα reproduce the 120 ordering as expected. α = 1, 2, 3, of the Lagrange multiplier. Finite Lagrange± mul- ◦ tipliers λ (1) , in turn induce two finite secondary Fourier 2qα ± ˆ α (2) 2. Vector chirality and vortices components Ω (2) with qα,β = Q t (2aβ aα) where Z2 qα,β − − β = α and so on. 6In the following, we discuss a Luttinger-Tisza approxima- It turns out that the approximate classical ground state (13) tion where we limit ourselves to the lowest finite Fourier com- corresponds to a triangular lattice of condensed Z2 vortices, ˆ α ˆ α thus confirming the numerical results of Ref. 20. This is best ponents Ω (1) and Ω (2) for the spin and λ0 and λ (1) q q 2qα α α,β ± seen by defining chirality vectors on upward pointing triangles for the Lagrange multiplier; all higher Fourier modes are ne- of the lattice glected. In principle, this approximation can be systematically improved by including higher order modes. Minimizing the κ(r) = 2 S S + S S + S S
. 3√3 r × r+ax r+ax × r+ay r+ay × r functional (10) within this approximation we obtain for the (14) energy per site The length of κ(r) measures the rigidity of the local 120◦ or- S2 h  π + 6t π 3t 22 ε (t) = J cos + 17 sin − (12) dering and it vanishes at the center of each Z2 vortex . The LT − 9 H 3 6 chirality vector profile, that derives from Eq. (13), is shown in π + 6t π + 15t Fig.3 and clearly reveals the vortex crystal. + 8 sin + sin Z2 6 6 Within our Luttinger-Tisza approximation we find three  π + 6t π + 6ti zero modes for the Z2 vortex crystal represented by the phase + JK cos + 8 sin , 3 6 φ and the vector r0. The latter are expected as the vortex crys- tal spontaneously breaks translational symmetry so that a con- which still depends on the parameter t that quantifies the dis- stant shift of the origin r0 does not cost any energy. The corre- tance of the primary Bragg peak from the corner of the Bril- sponding low-energy excitations are just the effective acous- (1) louin zone, qα = Q t aα. The value of tmin identifying tic phonon excitations of the vortex crystal. If the coupling − the position of the minimum of the function (12) finally de- between the two-dimensional atomic triangular lattice planes termines the ground state energy εLT(tmin). This analytical of Ba3IrTi2O9 is sufficiently small, these low-energy modes estimate for the ground state energy is found to be in excel- will destroy true long-range order of the Z2 vortex crystal at lent agreement with numerical estimates obtained from Monte any finite temperature that will be reflected in a characteris- Carlo simulations discussed in Sec.IVC. tic broadening of the Bragg peaks in the structure factor of The corresponding state is given by Fig.2 b). 4S n Sγ (r) Re eiφ (13) ≈ 3√3 × C. Polarized neutron scattering  o i(Q taγ ) (r r ) 1 X i(Q t(2aη aγ )) (r r ) e − · − 0 + e − − · − 0 , 4 η=γ The structure factor of the Z2 vortex crystal possesses as a 6 hallmark of the Kitaev interaction a characteristic correlation where Sγ is the γ-component of the spin and the ground state between the positions of the Bragg peaks and the associated is obtained by setting t = tmin. The first term in Eq. (13) spin-components, see Fig.2 b). We suggest to resolve this is the most important, primary Fourier component which also correlation with the help of spherical neutron polarimetry. possesses the smallest deviation of momentum from the cor- The probability that an incoming neutron with spin σin ner of the Brillouin zone, Q. The secondary Fourier compo- is scattered into a spin-state σout is given by the energy- nents have a smaller weight and are shifted further away by integrated scattering cross section σσout,σin (q), where q is the 5

dual 120 order nematic phase dual Z2 vortex crystal Z2 vortex crystal

FM 120 order

Figure 3. (Color online) Z2 vortex crystal stabilized for JH > 0 Z6 FM dual Z6 FM in the presence of a small but finite Kitaev interaction JK revealed by the chirality vectors of Eq. (14) which were computed from the dual O(3) FM classical ground state (13) in the Luttinger-Tisza approximation. The colour code shows the length of the chirality vector, |κ(r)|, normal- Figure 4. (Color online) Phase diagram of the Hamiltonian (1) with ized to one, that becomes minimal at the Z2 vortex cores. The arrows parametrization (JH ,JK ) = (cos α, sin α) as obtained from exact in the close-up of the left panel correspond to projections of κ(r) diagonalization data. Solid lines show the mapping between two onto the x-y plane. Klein-dual points. Red lines mark the location of the four SU(2)- symmetric points. Yellow diamonds mark the two Kitaev points. transfered momentum. Consider a polarizer and analyzer with an orientation specified by the unit vectors ein and eout, re- a) b) spectively. The total probability and the relative probability that a neutron is detected with spin e is then given by ± out X FM dual FM 120° dual 120° σ(q, eout, ein) = στoutσout,σin (q), (15) τ = 1 out ± X c) ∆σ(q, eout, ein) = τoutστoutσout,σin (q), (16) τ = 1 out ± Klein respectively. The polarization is then defined by the ratio duality P(q, eout, ein) = ∆σ(q, eout, ein)/σ(q, eout, ein). In the following, we concentrate on the magnetically ordered phase when the scattering probabilities are dominated by magnetic Z2 vortex crystal dual Z2 vortex crystal Bragg scattering so that we can neglect all nuclear contribu- (only one sublattice shown) (only one sublattice shown) tions. For the particular choice that the axis of polarizer and analyzer coincide, eout = ein e, but are orthogonal to the Figure 5. (Color online) (a) and (b) Spin configurations for the four transfered momentum e q,≡ the polarization attributed to SU(2) symmetric points of the HK model (1). The gray diamonds in- magnetic scattering simplifies⊥ to37,38 dicate the unit cells of the order. (c) Snapshots of spin configurations in the Z2-vortex crystal (left) and its dual Z2-vortex crystal (right). q For clarity, only one of the three sublattices of the triangular lattice P eiχij( )ej mag(q, e, e) = 2 1, (17) is shown.Yellow arrows point upwards out of the plane, while blue ˆe q χkl(q)(δkl qˆkqˆl) − ⊥ − arrows point downwards out of the plane. q where ˆq = q is the orientation of momentum and χij(q) = | | χij(q, ω = 0) is the spin susceptibility at zero frequency, Bragg peak, one expects for e = ˆz the value Pmag = 1 in P Z contrast to mag = 1 that is obtained for e in the direction ∞ iωt − χij(q, ω) = i dt e [Si(q, t), Sj( q, 0)] . (18) perpendicular to ˆz and q. A systematic variation of the an- 0 h − i alyzer/polarizer orientation e should therefore allow, in prin- ciple, to resolve the correlation between the diagonal compo- The magnetic structure factor of the vortex crystal, that Z2 nents χ and their Bragg peak position. follows from Eq. (13), has only non-zero diagonal compo- ii nents, χii, which however differ from each other and, more- over, possess different Bragg peak positions. For example, for our choice of the Kitaev interaction the χzz component is ex- IV. FULL PHASE DIAGRAM (1) pected to exhibit a primary Bragg peak at q = Q taz = 1 4π 1 √3 − a ( 3 (1, 0, 0) t( 2 , 2 ), 0) where a is the lattice constant After a detailed discussion of the magnetic structure close and we assumed− for− simplicity− that the two-dimensional trian- to the antiferromagnetic Heisenberg point in the previous sec- gular lattice lies in the x-y plane. Measuring at this particular tion, we now turn to the remaining part of the phase diagram. 6

It is represented in Fig.4 by a circle with the help of the a) z b) parametrization (JH ,JK ) = (cos α, sin α). 0.006 Importantly, the HK model (1) exhibits a duality6,25 (also referred to as the Klein duality14) relating a pair of interactions JK on the right-hand side of the circle to a pair of interactions on 0.000 the left-hand side, i.e. JH JH and JK 2JH + JK . x y The corresponding dual states→ are − related by a→ four-sublattice basis transformation, see appendixA for more explanations. As a consequence, the antiferromagnetic, α = 0, as well as Figure 6. (Color online) a) Ground state energy of the quantum the ferromagnetic Heisenberg point, α = π, both possess a model for ferromagnetic JH < 0 in an external Zeeman field as a dual giving rise to four SU(2) symmetric points marked by red function of the direction of the applied magnetic field B, where we bars in Fig.4. In particular, this maps the ferromagnetic state have subtracted the ground state energy for B = |B|ˆz. The Kitaev coupling strength is JK /|JH | = tan(11π/10) ≈ 0.32. The en- for JH < 0 at α = π to a dual ferromagnet at JH > 0 and J < 0 consisting of alternating strips of up and down point- ergy is minimal when the magnetization is pinned along one of the K three axes, and maximal when pointing along the space diagonals. ing spins, see Fig.5 a). Similarly, the 120 ◦ ordered state and b) The same results shown for the cut along the yellow line in a). its surrounding -vortex crystal phase around the J > 0 Z2 H Each line in b) corresponds to a different value of JK /JH . While Heisenberg point map to a dual phase in the upper left quad- for JK = 0 the ground state energy is independent of the direction rant with JH < 0 and JK > 0 with the respective orderings of the magnetic field, the directional dependence becomes increas- illustrated in Fig.5 b) and c). ingly pronounced upon increasing JK /|JH |. The dashed line is a fit In the following, we first elaborate in Sec.IVA on the fer- of Eq. (20). romagnetic phase and the influence of a finite Kitaev inter- action on the order parameter space. Second, in Sec.IVB we examine the physics close to the Kitaev point α = π/2 For JK = 0, this indeed corresponds to the exact ground where the classical ground state manifold is macroscopically state energy. Any finite JK , however, gives rise to fluctua- degenerate so that quantum fluctuations have a profound ef- tion corrections to the ground state energy that also discrimi- fect. Third, in Sec.IVC we finally discuss the ground state nate between the various orientations of Ωˆ. The leading 1/S- energies of the classical as well as of the quantum model that fluctuation correction to the energy is computed in appendix lead to the phase diagram in Fig.4 B and reads in lowest order in the Kitaev interaction JK

2 S JK 3(2√3 π) 4 4 4 δεFM = − 1 + Ωˆ + Ωˆ + Ωˆ . (20) Z − 2 J 8π x y z A. 6 ferromagnet | H | This correction favors the vector Ωˆ to point along one of the At the Heisenberg point JH < 0 and JK = 0, the ex- six equivalent 100 directions (as 2√3 π > 0). Whereas act ground state of the Hamiltonian is the ferromagnetic spin- h i − at the Heisenberg point, JK = 0, the ferromagnetic ground configuration where the order parameter is allowed to cover state manifold is the full sphere, S2, a finite Kitaev interaction the whole sphere S2, i.e., to point in any direction. In the reduces this manifold to only six points corresponding to a Z6 presence of a finite JK , however, fluctuations discriminate be- ferromagnetic order parameter. tween the various orientations of the ferromagnetic order pa- rameter and reduce the order parameter space from the sphere to Z6, i.e., to only six points. A similar order-by-disorder 2. Numerical evidence mechanism has recently been discussed27 with regard to dis- tortions in the hexagonal HK model. To corroborate our analytical results for the reduced order We concentrate here on the regime of the phase diagram parameter space for J = 0 around the ferromagnetic Heisen- adjacent to the ferromagnetic Heisenberg point (dark blue K berg point, we performed6 exact diagonalization calculations shaded in Fig.4). With the help of the duality transforma- on small systems. We implemented lattice clusters with peri- tion analogous conclusions then apply to the dual ferromagnet odic boundary conditions containing 12 sites, with a geometry corresponding to the light blue shaded regime in Fig.4. that preserves the C6 rotational symmetry of the triangular lat- tice. By applying a small magnetic field B to each spin,

1. Analytical arguments cos(φ) sin(θ) B = B sin(φ) sin(θ) , (21) The classical ferromagnetic ground state is given by a con- cos(θ) stant, homogeneous spin configuration, Ωˆ(r) Ωˆ with Ωˆ 2 = ≡ where φ [0, 2π) and θ [0, π], the magnetization was 1. The corresponding classical energy per site is independent ∈ ∈ of the orientation of Ωˆ and reads forced to point in different directions. Fig.6 a) shows re- sults for the change in the ground state energy as a func-   tion of the orientation of B with respect to the parallel align- ε = S2 3J + J (19) FM H K ment B ˆz for a small finite Kitaev coupling J / J = k K | H | 7

a) b)

Figure 8. (Color online) Energy gaps of a 3 × L triangular lattice Figure 7. (Color online) Histogram of the spin expectation value ob- strip with open boundary conditions. All values are given in relation tained with the help of finite-temperature Monte Carlo simulations of to the ground state energy E0, i.e. ∆E1 = E1 − E0. The figures x x the classical HK model close to the ferromagnetic Heisenberg point. on the right show numerical results for hSr Sr i spin correlations, 2 0 Whereas for JK = 0 in panel (a) the spin covers the full S sphere, where the black disk with the white dot indicates the position r0, the thermal fluctuations in the presence of a finite JK 6= 0 favor the diameter of the disks indicates the strength of the correlation and the alignment along one of the six h100i directions. color indicates the sign, with red corresponding to negative (antifer- romagnetic) and black to positive (ferromagnetic) correlations. For details, see the main text. tan(11π/10) 0.32. In agreement with our analysis above, the ground state≈ energy of the system is minimal when the magnetization points along one of the six 100 directions. later reference, the energy per site of the classical ground state Scanning the orientation of B along the yellowh linei shown in close to the antiferromagnetic Kitaev point is given by

Fig.6 a), we compare in Fig.6 b) the effect of different Ki- 2 εnematic = S (JH + JK ). (22) taev couplings (solid lines). While for JK = 0 the energy is − independent of the orientation of B, for any finite J = 0 K In order to investigate this nematic ordering of the quantum the energy immediately acquires an orientational dependence,6 model, we calculated the energies of the ground state and the that becomes more pronounced as J increases. The black K first few excited states using the density matrix renormaliza- dashed line in Fig.6b) is a fit of Eq. (20), showing perfect tion group (DMRG)26,33. Once the ground state was found, agreement. we targeted excited states by successively calculating states The same reduction of the order parameter space is al- of lowest energy that are orthogonal to all previously found ready at work on the classical level. Fig.7 shows result of states. While the DMRG is highly successful for 1D systems, a finite-temperature Monte Carlo simulation of the classical it can also be extended to systems with a small finite width, HK model. Whereas for J = 0 the order parameter covers K and we considered triangular lattice systems of width 3 and the S2 sphere uniformly as illustrated in Fig.7 a), the thermal 4 and varying length with open boundary conditions. We ran fluctuations in the presence of a finite J favor the alignment K calculations at bond dimensions M = 600, 800, 1000 making of the order parameter along one of the six 100 directions as sure that the energies converged. shown in Fig.7 b). h i The geometry of the considered lattice clusters breaks the C6 symmetry of the lattice and the spins order antiferromag- netically in the spin component corresponding to the interac- B. Nematic order close to the Kitaev point tion term along the longer direction. In Fig.8 we show the energy differences between the lowest 8 excited states and the In the classical limit, the Kitaev model on the triangular ground state, alongside spin-spin correlators. The first three lattice possesses a macroscopic ground state degeneracy as excited states collapse exponentially onto the ground state pointed out in Ref. 20. The spins form anti- or ferromagneti- energy as the length of the system increases. Likewise, the cally ordered Ising chains, for JK > 0 and JK < 0, respec- next four excited states collapse to the same energy, however tively, along one of the three lattice directions. The Kitaev growing linearly in system length. From the calculated spin- interaction, however, does not couple the ordering of the indi- spin correlators we can identify this excitation to be given by vidual chains thus giving rise to a 3 2L-fold sub-extensive a breaking of the antiferromagnetic ordering between next- ground state degeneracy where L is× the linear system size. nearest neighbor chains. Finally the 8th excited level cor- Each ground state breaks the combined symmetry of the HK responds to a local defect in a chain, which is indicated by Hamiltonian of a C6 lattice rotation and a cyclic spin exchange the vanishing spin correlation in the center left corner of the so that the ordering is that of a spin nematic. While the ferro- lattice cluster. Fig.9 shows the spin-spin correlations in the magnetic Kitaev point, JK < 0, only separates the ferromag- ground states for systems of width 3 and 4 at the antiferromag- netic and the dual ferromagnetic order, which is immediately netic Kitaev point (JH = 0). While nearest neighbor chains stabilized for any finite JH , an extended nematic phase arises are uncorrelated, there is a clear antiferromagnetic correla- 20 close to the antiferromagnetic Kitaev point, JK > 0 . For tion between next-nearest neighbor chains in the spin compo- 8

a) quantum model

b) classical model

2 2 Figure 10. (Color online) Upper panel (a): Ground state energy E0 (black) and its second derivative, −d E0/dα , (red) for the HK quantum model obtained from exact diagonalization of small clusters. Peaks in the second derivative indicate the position of phase transitions. The black and red arrows indicate the corresponding axis for each data set. Lower panel (b): Classical energies (gray dots, Monte Carlo) and quantum energies (dashed, ED). The colored solid lines show analytical estimates for the classical ground state energies of the respective ordered phases, namely, Eq. (B1) for the Z6 ferromagnet and its dual, Eq. (12) after minimization for the Z2-vortex crystal and its dual and Eq. (22) for the nematic phase. The classical and quantum energies touch at the two fluctuation free points: the Heisenberg FM at α/π = 1 and its dual point. The upper and lower ring summarize in the spirit of Fig.4 the extension of the various phases of the quantum and the classical model, respectively.

3-leg ladder 4-leg ladder nent given by the chain direction. This mechanism locks the spin alignment of next-nearest neighbor chains to each other SxSx h r r0 i and thus reduces the macroscopic degeneracy of the ground state from 3 2L to the non-extensive value 3 22. Other spin components× show only very short-ranged correlations× as SySy h r r0 i shown in the lower two panels of Fig.9. Upon including a non-vanishing Heisenberg interaction correlations also form between nearest-neighbor chains further lifting the degener- SzSz h r r0 i acy to 3 2 states (not shown), which however preserve the nematic nature× of the Kitaev point. Figure 9. (Color online) Spin-spin correlations in the ground state of the antiferromagnetic Kitaev model on the triangular lattice. Black γ γ circles indicate positive correlations, hSi Sj i > 0, whereas the red circles denote negative correlations. The small white dot indicates the position r0. The geometry of the lattice clusters lifts the degen- eracy of the lattice direction, favoring chains antiferromagnetically coupled with their x-component along the x-direction while correla- tions along the y- and z-directions are suppressed. Whereas adjacent chains remain uncoupled, next-nearest neighbor chains couple anti- ferromagnetically.

C. Phase boundaries and ground state energies 6 4 = 24 lattice sites and periodic boundary conditions as× well as clusters with 27 lattice sites keeping the original The phase boundaries in Fig.4 have been determined by calculating the ground state energy for clusters with N = 9

34 C3 lattice symmetry – with both clusters preserving the SU(2) on the ALPS libraries . The numerical simulations were per- symmetry of the Heisenberg points under the Klein duality. formed on the CHEOPS cluster at RRZK Cologne. Some Using exact diagonalization (ED) techniques, we have deter- of the figures were created using the Mayavi library35 and mined the phase boundaries by identifying the points where Vesta36, respectively. the second derivative d2E/dα2 appears to diverge (on these finite systems), see the− upper panel of Fig. 10. For completeness, we have also repeated the Monte Carlo Appendix A: Klein duality transformation simulations of the classical model that were already per- formed in Ref. 20. The result for the classical ground state We review the Klein duality relating couplings on the left energies is shown in the lower panel of Fig. 10 together with and right-hand side of the circle phase diagram, see Fig. 11 b). a comparison to the ground state energies obtained from ED Under this transformation, the Heisenberg-Kitaev Hamilto- of the quantum model. As expected the two agree for the nian retains the same structure but the coupling parameters ferromagnetic Heisenberg model and its dual point indicat- change as ing the absence of quantum fluctuations around their classical ground states. We also compare the Monte Carlo data with JH JH ,JK 2JH + JK . (A1) the analytical estimates for the classical ground state energies → − → (colored solid lines), which approximate well the numerical The transformation is performed by dividing the triangular lat- result. It should be noted that the phase diagram for the quan- tice into four sublattices as illustrated in Fig. 11 a). Subse- tum HK model closely mimics the one found for the classical quently, each spin is subjected to a basis rotation, where the 20 HK model , which is due to the mainly classical nature of the spins on the sublattice labeled “id” are not changed. For the various ordered phases. The exceptions are the Kitaev points three remaining sublattices each spin is rotated by π around where quantum fluctuations have a profound effect and lift the the spin axis according to the sublattice labeling. Since a π macroscopic degeneracy of the ground state. rotation around one spin axis effectively inverses the sign of the two other components, we can write the full transforma- tion as V. CONCLUSIONS id :(Sx,Sy,Sz) ( Sx,Sy,Sz) (A2a) x y z → x y z To summarize, we propose that a Z2-vortex crystal phase x :(S ,S ,S ) ( S , S , S ) might be observed in the recently synthesized Ba IrTi O 18. → − − 3 2 9 y :(Sx,Sy,Sz) ( Sx,Sy, Sz) The latter forms a j = 1/2 Mott insulator, whose low- → − − z :(Sx,Sy,Sz) ( Sx, Sy,Sz). energy physics we argue to be captured by a Heisenberg- → − − Kitaev model on a triangular lattice. We reemphasize that the Since this transformation is a simple local rotation of the Z2-vortex crystal arises in the vicinity of the antiferromag- netic Heisenberg model, i.e. in the limit of small Kitaev inter- spin basis, the original Hamiltonian and its counterpart after actions, and thus in the experimentally most relevant parame- the transformation effectively describe the same physics, al- ter regime – as revealed by numerous microscopic studies12,13 beit for a resized unit cell. Interestingly, this transformation of the honeycomb Iridates indicating the presence of Kitaev- maps the SU(2) symmetric ferromagnetic and antiferromag- netic Hamiltonians at J = 0 and J = 1 onto Heisenberg- type interactions only in addition to a dominant Heisenberg K H ± 18 Kitaev Hamiltonians with JK = 2JH , revealing two more exchange. Initial samples of Ba3IrTi2O9 appear to suffer − from significant Ir-Ti site inversion obscuring the formation of SU(2) symmetric points in the phase diagram. These points any ordered phase, but better samples should exhibit a distinct signature in polarized neutron scattering as we have discussed J in detail. The physics of the triangular HK model is also rel- a) b) K evant to the honeycomb Iridates, for which it has been argued id that a next-nearest neighbor exchange (along the two triangu- x ↵ lar sublattices of the honeycomb lattice) is indeed present in y 28–31 JH the actual materials . Finally, we have left it to future re- z search to explore whether the Z2-vortex crystal also plays out 32 in the bilayer triangular lattice material Ba3TiIr2O9 , which is closely related to the Ba3IrTi2O9 compound by replacing the role of Ir and Ti. Figure 11. (Color online) a) 24 site cluster with periodic bound- ary conditions containing all symmetries except for the rotational C3 symmetry. The different symbols for the lattice sites indicate the four sublattices needed in the basis transformation underlying the Klein ACKNOWLEDGMENTS duality (A1). b) Circle parametrization of the Heisenberg-Kitaev in- teractions JH = J cos α and JK = J sin α with the magenta lines We acknowledge insightful discussions with M. Daghofer, indicating points on the left and right-hand side of the circle related L. Fritz, G. Jackeli, M. Punk, A. Rosch, and I. Rousochatza- by the Klein duality (A1). The filled yellow and green circles indi- kis. Our DMRG33 and exact diagonalization codes are based cate the points at which the Hamiltonian (1) is SU(2) symmetric. 10 and their corresponding phases are termed the “stripy” (anti- rections to the ground state that also discriminate between the )ferromagnets, due to the magnetic order after the basis rota- various orientations of Ωˆ. Performing a standard Holstein- tion. The spin configurations at these points are illustrated in Primakoff transformation, the spin-operator along the local z- Fig.5. axis, here defined by the classical vector Ωˆ, can be expressed ˜z as Si = S ai†ai where ai is a bosonic annihilation operator at the site i−. Moreover, Appendix B: Fluctuation correction to the ferromagnetic q q + ground state energy S˜ = 2S a†a a , S˜− = a† 2S a†a (B2) i − i i i i i − i i ˜ ˜x ˜y The classical ferromagnetic ground state is given by a con- where Si± = Si iSi . The spin-operator S within the labo- ± ˜ ˜ stant, homogeneous spin configuration, Ωˆ(r) Ωˆ with Ωˆ 2 = ratory frame is related to S by a rotation S = RS where 1. The corresponding classical energy per site≡ is independent  sin φ cos θ cos φ sin θ cos φ  of the orientation of Ωˆ and reads − − R =  cos φ cos θ sin φ sin θ sin φ  . (B3) − 2  0 sin θ cos θ εFM = S 3JH + JK (B1) and Ωˆ = R(0, 0, 1)T . Expanding the Hamiltonian in second For J = 0, this indeed corresponds to the exact ground state order in the bosonic operators one obtains = Nε + (2) K H FM H energy. Any finite JK , however, gives rise to fluctuation cor- with

  h i (2) 1 X ak S X X X ˆ 2 = (ak† a k)h(k) 2JH (cos(k aγ ) 1) + JK (cos(k aγ ) 1)(1 Ωγ ) H 2 − a† k − 2 · − · − − k 1.BZ − k 1.BZ γ=x,y,z γ=x,y,z ∈ ∈ (B4) where

h X n  + + 2 + 2  h(k) = S 2J (cos(k a ) 1)1 + 2J (cos(k a ) 1) e e−1 + (e ) σ + (e−) σ− (B5) H · γ − K · x − x x x x γ=x,y,z  + + 2 + 2  + x oi + (cos(k a ) 1) e e−1 + (e ) σ + (e−) σ− + (cos(k a ) 1)e e−(1 σ ) · y − y y y y · z − z z −

with the Pauli matrices σx, σy, and σz, and we used the ab- pute the correction to the classical ground state energy (B1). 1 T 1 x y breviations e± = R(1, i, 0) and σ± = (σ iσ ). In order to elucidate the analytical structure, we concentrate √2 ± 2 ± With the help of a Bogoliubov transformation we can com- on the contribution to this correction only of lowest order in the Kitaev interaction,

1 X h21(k)h12(k) δεFM = (B6) −4N h11(k) J =0 k 1.BZ K ∈ | 2 k a + 2 k a + 2 k a + 2 S JK X (cos( x) 1)(ex ) + (cos( y) 1)(ey ) (cos( z) 1)ez ez− = · − P · − − · − . −2N JH (1 cos(k aγ )) k 1.BZ γ=x,y,z | | ∈ − ·

To evaluate this expression we need the following integrals Here, we evaluated the integrals in the thermodynamic limit over the Brillouin zone where the volume of the first Brillouin zone is given by

1 X (cos(k aα) 1)(cos(k aβ) 1) P · − · − (B7) N (1 cos(k aγ )) k 1.BZ γ=x,y,z − · ∈ Z N 1 (cos(k aα) 1)(cos(k aβ) 1) →∞ dk · − · − −→ P (1 cos(k a )) V1.BZ 1.BZ γ=x,y,z − · γ 6√3 2π 5π 6√3 = − δ + − (1 δ ). 3π αβ 6π − αβ 11

2 = 8π using the identities The fluctuation correction to the energy in lowest order in the 1.BZ √3 V Kitaev interaction finally assumes the form given in Eq. (20). + 2 + 2 + 2 (ex ex−) + (ey ey−) + (ez ez−) = (B8) + 2 + 2 + 2 + 2 + (e e−) (e e−) + ((e ) + (e ) )e e− − x y − y x x y z z + 2 2 + ez ez−((ex−) + (ey−) ) 1  = 1 + Ωˆ 4 + Ωˆ 4 + Ωˆ 4 . 4 x y z

1 W. Witczak-Krempa, G. Chen, Y. B. Kim, and L. Balents, Cor- Foyevtsova, H. O. Jeschke, P. Gegenwart, and R. Valent´ı, Origin related Quantum Phenomena in the Strong Spin-Orbit Regime, of the insulating state in honeycomb iridates and rhodates, Phys. Annual Review of Condensed Matter Physics 5, 57 (2014). Rev. B 88, 035115 (2013); C. H. Sohn et al., Mixing between 2 1 3 B. J. Kim, Hosub Jin, S. J. Moon, J.-Y. Kim, B.-G. Park, C. S. Jeff = 2 and 2 orbitals in Na2IrO3: A spectroscopic and den- Leem, Jaejun Yu, T. W. Noh, C. Kim, S.-J. Oh, J.-H. Park, V. Du- sity functional calculation study, Phys. Rev. B 88, 085125 (2013). rairaj, G. Cao, and E. Rotenberg, Novel Jeff=1/2 Mott State In- 13 A. Shitade, H. Katsura, J. Kunes,ˇ X.-L. Qi, S.-C. Zhang, and N. duced by Relativistic Spin-Orbit Coupling in Sr2IrO4, Phys. Rev. Nagaosa, Quantum Spin Hall Effect in a Transition Metal Ox- Lett. 101, 076402 (2008). ide Na2IrO3, Phys. Rev. Lett. 102, 256403 (2009); H.-C. Jiang, 3 B. J. Kim, H. Ohsumi, T. Komesu, S. Sakai, T. Morita, H. Takagi, Z.-C. Gu, X.-L. Qi, and S. Trebst, Possible proximity of the and T. Arima, Phase-Sensitive Observation of a Spin-Orbital Mott Mott insulating iridate Na2IrO3 to a topological phase: Phase State in Sr2IrO4, Science 323, 1329 (2009). diagram of the Heisenberg-Kitaev model in a magnetic field, 4 Yogesh Singh, S. Manni, J. Reuther, T. Berlijn, R. Thomale, W. Phys. Rev. B 83, 245104 (2011); J. Reuther, R. Thomale, and Ku, S. Trebst, and P. Gegenwart, Relevance of the Heisenberg- S. Trebst, Finite-temperature phase diagram of the Heisenberg- Kitaev Model for the Honeycomb Lattice Iridates A2IrO3, Phys. Kitaev model, Phys. Rev. B 84, 100406(R) (2011); C. C. Price Rev. Lett. 108, 127203 (2012). and N. B. Perkins, Critical Properties of the Kitaev-Heisenberg 5 D. Pesin and L. Balents, Mott physics and band topology in mate- Model, Phys. Rev. Lett. 109, 187201 (2012); S. Bhattacharjee, S.- rials with strong spinorbit interaction, Nature Phys. 6, 376 (2010). S. Lee, and Y. B. Kim, Spinorbital locking, emergent pseudo-spin 6 G. Khaliullin, Orbital Order and Fluctuations in Mott Insulators, and magnetic order in honeycomb lattice iridates, New J. Phys. Prog. Theor. Phys. Suppl. 160, 155 (2005). 14, 073015 (2012); J. Chaloupka, G. Jackeli, and G. Khaliullin, 7 G. Jackeli and G. Khaliullin, Mott Insulators in the Strong Spin- Zigzag Magnetic Order in the Iridium Oxide Na2IrO3, Phys. Rev. Orbit Coupling Limit: From Heisenberg to a Quantum Compass Lett. 110, 097204 (2013); C. Price and N. B. Perkins, Finite- and Kitaev Models, Phys. Rev. Lett. 102, 017205 (2009). temperature phase diagram of the classical Kitaev-Heisenberg 8 J. Chaloupka, G. Jackeli, and G. Khaliullin, Kitaev-Heisenberg model, Phys. Rev. B 88, 024410 (2013). Model on a Honeycomb Lattice: Possible Exotic Phases in Irid- 14 I. Kimchi and A. Vishwanath, Kitaev-Heisenberg models for ium Oxides A2IrO3, Phys. Rev. Lett. 105, 027204 (2010). iridates on the triangular, hyperkagome, kagome, fcc, and py- 9 K. I. Kugel and D. I. Khomski˘ı, The Jahn-Teller effect and mag- rochlore lattices, Phys. Rev. B 89, 014414 (2014). netism: transition metal compounds, Soviet Physics Uspekhi 25, 15 E. K.-H. Lee, R. Schaffer, S. Bhattacharjee, and Y. B. Kim, 213 (1982). Heisenberg-Kitaev model on the hyperhoneycomb lattice, Phys. 10 A. Kitaev, Anyons in an exactly solved model and beyond, Ann. Rev. B 89, 045117 (2014); M. Hermanns and S. Trebst, Quan- Phys. 321, 2 (2006). tum spin liquid with a Majorana Fermi surface on the three- 11 S. K. Choi et al., Spin Waves and Revised Crystal Structure dimensional hyperoctagon lattice, Phys. Rev. B 89, 235102 of Honeycomb Iridate Na2IrO3, Phys. Rev. Lett. 108, 127204 (2014); I. Kimchi, J. G. Analytis, and A. Vishwanath, Three- (2012); R. Comin et al., Na2IrO3 as a Novel Relativistic Mott In- dimensional quantum spin liquids in models of harmonic- sulator with a 340-meV Gap, Phys. Rev. Lett. 109, 266406 (2012); honeycomb iridates and phase diagram in an infinite-D approx- Feng Ye, Songxue Chi, Huibo Cao, Bryan C. Chakoumakos, imation, Phys. Rev. B 90, 205126 (2014); M. Hermanns, K. Jaime A. Fernandez-Baca, Radu Custelcean, T. F. Qi, O. B. Kor- O’Brien, and S. Trebst, Weyl Spin Liquids, Phys. Rev. Lett. 114, neta, and G. Cao, Direct evidence of a zigzag spin-chain structure 157202, (2015). in the honeycomb lattice: A neutron and x-ray diffraction investi- 16 T. Takayama, A. Kato, R. Dinnebier, J. Nuss, and H. Takagi, gation of single-crystal Na2IrO3, Phys. Rev. B 85, 180403 (2012); Three-dimensional quantum spin liquids in models of harmonic- H. Gretarsson et al., Crystal-Field Splitting and Correlation Ef- honeycomb iridates and phase diagram in an infinite-D approxi- fect on the Electronic Structure of A2IrO3, Phys. Rev. Lett. 110, mation Phys. Rev. Lett. 114, 077202 (2015). 076402 (2013); H. Gretarsson et al., Magnetic excitation spec- 17 K. A. Modic et al., Realization of a three-dimensional trum of Na2IrO3 probed with resonant inelastic x-ray scattering, spinanisotropic harmonic honeycomb iridate, Nature Comm. 5, Phys. Rev. B 87, 220407 (2013). 4203 (2014). 12 I. I. Mazin, H. O. Jeschke, K. Foyevtsova, R. Valent´ı, and D. I. 18 T. Dey, A. V. Mahajan, P. Khuntia, M. Baenitz, B. Koteswararao, Khomskii, Na2IrO3 as a Molecular Orbital Crystal, Phys. Rev. and F. C. Chou, Spin-liquid behavior in Jeff=12 triangular lattice Lett. 109, 197201 (2012); K. Foyevtsova, H. O. Jeschke, I. I. compound Ba3IrTi2O9 Phys. Rev. B 86, 140405(R) (2012). Mazin, D. I. Khomskii, and R. Valent´ı, Ab initio analysis of the 19 For the convenience of the inclined reader we provide a VESTA tight-binding parameters and magnetic interactions in Na2IrO3, visualization file of the Ba3IrTi2O9 atomic structure in the auxil- Phys. Rev. B 88, 035107 (2013); I. I. Mazin, S. Manni, K. iary material of this arXiv submission. 12

20 I. Rousochatzakis, U. K. Rssler, J. van den Brink, M. Daghofer, 86, 155127 (2012). 30 Z2-vortex lattice in the ground state of the triangular Kitaev- J. Reuther, R. Thomale, and S. Rachel, Spiral order in the honey- Heisenberg model, arXiv:1209.5895 (2012). comb iridate Li2IrO3, Phys. Rev. B 90, 100405(R) (2014). 21 S. J. Miyake, Spin-Wave Results for the Staggered Magnetization 31 Y. Sizyuk, C, Price, P. Wolfle,¨ and N. B. Perkins, Importance of of Triangular Heisenberg Antiferromagnet, J. Phys. Soc. Jpn. 61, anisotropic exchange interactions in honeycomb iridates: Mini- 983 (1992); A. V. Chubukov, S. Sachdev and T. Senthil, Large-S mal model for zigzag antiferromagnetic order in Na2IrO3, Phys. expansion for quantum antiferromagnets on a triangular lattice,J. Rev. B 90, 155126 (2014) Phys. Condens. Matter 6, 8891 (1994). 32 T. Sakamoto, Y. Doi, and Y. Hinatsu, Crystal structures and 22 H. Kawamura and S. Miyashita, Phase Transition of the Two- magnetic properties of 6H-perovskite-type oxides Ba3MIr2O9 Dimensional Heisenberg Antiferromagnet on the Triangular Lat- (M=Mg, Ca, Sc, Ti, Zn, Sr, Zr, Cd and In), J. Solid State Chem. tice, J. Phys. Soc. Jap. 53, 4138 (1984). 179, 2595 (2006). 23 The Supplementary Material is provided in the appendices. 33 M. Dolfi, B. Bauer, S. Keller, A. Kosenkov, T. Ewart, A. Kantian, 24 J.M. Luttinger and L. Tisza, Theory of Dipole Interaction in Crys- T. Giamarchi, and M. Troyer, Matrix product state applications tals, Phys. Rev. 70, 954 (1946). for the ALPS project, Comput. Phys. Commun. 185, 3430 (2014). 25 G. Khaliullin and S. Okamoto, Quantum Behavior of Orbitals in 34 B. Bauer et al., The ALPS project release 2.0: open source Ferromagnetic Titanates: Novel Orderings and Excitations, Phys. software for strongly correlated systems, J. Stat. Mech. P05001 Rev. Lett. 89, 167201 (2002). (2011). 26 S. R. White, Density matrix formulation for quantum renormal- 35 P. Ramachandran, and G. Varoquaux, Mayavi: 3D Visualization ization groups, Phys. Rev. Lett. 69, 2863 (1992). of Scientific Data, IEEE Computing in Science & Engineering 13, 27 E. Sela, H.-C. Jiang, M. H. Gerlach, and S. Trebst, Order-by- 40 (2011). disorder and spin-orbital liquids in a distorted Heisenberg-Kitaev 36 K. Momma and F. Izumi, VESTA 3 for three-dimensional visu- model, Phys. Rev. B 90, 035113 (2014); S. Nishimoto et al., alization of crystal, volumetric and morphology data, J. Appl. Strongly frustrated triangular spin lattice emerging from triplet Crystallogr., 44, 1272-1276 (2011). 37 dimer formation in honeycomb Li2IrO3, arXiv:1403.6698. M. Blume, Polarization Effects in the Magnetic Elastic Scattering 28 M. Kargarian, A. Langari, and G. A. Fiete, Unusual magnetic of Slow Neutrons, Phys. Rev. 130, 1670 (1963). phases in the strong interaction limit of two-dimensional topolog- 38 S. Maleyev, V. Baryakhtar, and R. Suris, The Scattering of Slow ical band insulators in transition metal oxides, Phys. Rev. B 86, Neutrons by Complex Magnetic Structures, Fiz. Tverd. Tela 4, 205124 (2012). 3461 (1962); English translation in Sov. Phys. Solid State 4, 2533 29 J. Reuther, R. Thomale, and S. Rachel, Magnetic ordering phe- (1963). nomena of interacting quantum spin Hall models, Phys. Rev. B