Quantum phase diagram of spin-1 J1 − J2 Heisenberg model on the square lattice: an infinite projected entangled-pair state and density matrix renormalization group study R. Haghshenas1,∗ Wang-Wei Lan2, Shou-Shu Gong3,y and D. N. Sheng1 1Department of Physics and Astronomy, California State University, Northridge, California 91330, USA 2Department´ de Physique, Universite´ de Sherbrooke, Quebec,´ Canada 3Department of Physics, Beihang University, Beijing, 100191, China We study the spin-1 Heisenberg model on the square lattice with the antiferromagnetic nearest-neighbor J1 and the next-nearest-neighbor J2 couplings by using the infinite projected entangled-pair state (iPEPS) ansatz and density matrix renormalization group (DMRG) calculation. The iPEPS simulation, which studies the model directly in the thermodynamic limit, finds a crossing of the ground state from the Neel´ magnetic state to the stripe magnetic state at J2=J1 ' 0:549, showing a direct phase transition. In the finite-size DMRG calculation on the cylinder geometry up to the cylinder width Ly = 10, we find that around the same critical point the Neel´ and the Stripe orders are strongly suppressed, which implies the absent of an intermediate phase. Both calculations identify that the stripe order comes with a first-order transition at J2=J1 ' 0:549. Our results indicate that unlike the spin-1=2 J1 − J2 square model, quantum fluctuations in the spin-1 model may not be strong enough to stabilize an intermediate non-magnetic phase. PACS numbers: 75.40.Mg, 75.10.Jm, 75.10.Kt, 02.70.-c I. INTRODUCTION scale. Classically, this model in the large S limit has a Neel´ and a stripe AFM phase separated at J2 = 0:5. After consider- Frustrated magnetic systems play a key role in understand- ing quantum fluctuations, the early studies based on the mod- 12,34 ing the exotic phases of matter1,2. As we know, frustra- ified spin-wave theory and Schwinger-Boson mean-field 35 tion can enhance quantum fluctuations by posing incompat- theory predicted that quantum fluctuations for the systems ibility on the local interaction energy to be simultaneously with spin magnitude S > 0:7 are not strong enough to sta- satisfied, which may destroy magnetic long-range order and bilize a non-magnetic intermediate phase. Thus, mean-field lead the systems into novel quantum phases, such as valence- results suggested a direct phase transition from the Neel´ to bond solid (VBS)3–5 and quantum spin liquid6–8. In addition, the stripe AFM phase for the spin-1 model. This result was quantum phase transitions between such phases may defy the later confirmed by the coupled cluster method, which found Ginzburg-Landau theory—referred to the deconfined quan- a first-order transition between the Neel´ and stripe phase at 36 tum criticality9—which makes the physics of frustrated quan- J2 ' 0:55 . Interestingly, the recent density matrix renor- 37 tum magnetism a fascinating subject both in theoretical and malization group (DMRG) study challenged this result: it experimental senses. Among the various frustrated antiferro- predicted a non-magnetic phase in the small intermediate re- 10–13 0:525 J 0:555 magnets, the spin-SJ1 − J2 square Heisenberg models gion for . 2 . , and suggested that this non- are well-known examples and have stimulated extensive the- magnetic phase might be continuously connected to the limit 38 oretical studies over the last two decades. The competing in- of the decoupled Haldane spin chains by tuning the spa- teractions in such systems may stabilize a non-magnetic in- cial anisotropic couplings J1x and J1y. Such a non-magnetic termediate phase if quantum fluctuations are not strongly sup- phase in spin-1 model is quite interesting not only because pressed10–13. One of the successful examples is the spin-1=2 it goes beyond the physics in the mean-field description, but J1 − J2 model, in which a non-magnetic intermediate phase also because it might be related to the nematic non-magnetic 26 has been identified by using different methods although the phase in the iron-based superconductor material FeSe . nature of the phase is still controversial14–20. In this article, our main goal is to reexamine the phase di- Meanwhile, the frustrated spin-1 square Heisenberg models agram of the spin-1 J1 − J2 SHM based on the variational (SHM) are also typical for studying frustrated magnetism. Re- tensor-network ansatz and DMRG simulation39. In previous cently, the studies on the magnetism of the iron-based super- studies, while the mean-field calculation may not fully con- conductors have drawn extensive interests in investigating the sider quantum fluctuations12,34,35, the DMRG simulation may arXiv:1802.00874v3 [cond-mat.str-el] 31 May 2018 novel quantum phases, particularly the non-magnetic phase have difficulty to pin down the intermediate region as a non- with lattice nematic order, in different spin-1 SHMs21–33. magnetic phase due to the finite-size effects37. To this end, we Among the various models, the spin-1 J1 − J2 SHM is prob- use the state-of-the-art numerical methods to systematically ably the most fundamental model, which is defined as study the model. We use the U(1)-symmetric infinite pro- jected entangled-pair state (iPEPS) ansatz40,41 and the SU(2)- X X H = J1 Si · Sj + J2 Si · Sj; symmetric finite-size DMRG to study the system from dif- hi;ji hhi;jii ferent limits: the iPEPS is directly applied in the thermody- namic limit, significantly diminishing possible finite-size ef- where hi; ji and hhi; jii denote the nearest-neighbor and the fects; SU(2)-symmetric DMRG obtains accurate results on next-nearest-neighbor pairs, and J1 and J2 are both antifer- finite-size system. In the iPEPS ansatz, the only control pa- romagnetic (AFM) couplings. We set J1 = 1 as the energy rameter is the so-called bond dimension D which controls en- 2 tanglement in the system. To simulate highly entangled states (larger bond dimensions), we implement U(1) symmetry into the iPEPS ansatz42. We expect that a close comparison of these different approaches could substantially improve our un- derstanding of the intermediate regime. In our iPEPS simulation, we use a hysteresis effect to ac- curately determine the quantum phase transition and its na- ture43. We initialize the iPEPS ansatz with different types of wavefunctions to find the lowest variational ground-state en- ergy. It is found that either the Neel´ or the stripe state pro- FIG. 1. (Color online) Tensor-network representation of a U(1)- vides the lowest iPEPS energy throughout the coupling pa- symmetric iPEPS j i. (a) The iPEPS j i is made of the U(1)- rameter range. The energy of the Neel´ and the stripe states invariant tensors fa; b; c; dg periodically repeated through the infi- cross each other at the critical point J2 ' 0:549, where the nite square lattice. (b) The scalar product h j i is calculated by magnetic order parameters (defined later) remain non-zero in using the so-called environment tensors fE1; ··· ;E12g obtained by the D ! 1 limit, suggesting a first-order phase transition CTMRG approach. The bond dimensions D; χ controls accuracy of between the two magnetic order phases. We also observe that the iPEPS anstaz. the correlation length is unlikely to show a divergent behav- ior around the transition point, further supporting a first-order transition. In our DMRG calculation with the improved sys- determines the number of the variational parameters, hence, tem size for size-scaling analysis, we find that the Neel´ order controlling the accuracy of the iPEPS ansatz. It also repre- could persist to J2 ' 0:545 and the stripe order grows up sents the amount of entanglement, so that by increasing it even sharply at J2 ' 0:55, showing a very narrow non-magnetic ‘highly entangled states’ could be accurately approximated45. regime, which is quite smaller than the previous DMRG re- 37 In this paper, as shown in Fig.1(a), we use (up to) a 2 × 2 sult ∼ 0:03 . The fast-shrinking intermediate regime, which unit cell iPEPS constructed from five-rank independent ten- is observed in the finite-size scaling with increased system sors fa; b; c; dg. We exploit U(1) symmetry, making all ten- size in the DMRG results, may suggest the vanishing non- sors fa; b; c; dg to be U(1) invariant, to reach the larger bond magnetic phase. The non-magnetic phase with a strong lattice dimensions up to D ∼ 946. We perform the optimization pro- nematicity in spin-1 models, which has been proposed for the 26 cedure by using imaginary-time evolution in the class of the nematic paramagnetic phase of FeSe , might be stabilized by iPEPS47 considering other compting and/or frustrating interactions. −τH The paper is organized as follows. We briefly discuss the j i+1i = e j ii; numerical methods and define the order parameters used in this paper in Sec.II. Our main numerical results are presented where j i+1i at each step i is represented by an iPEPS—τ in Sec. III. In Sec. III A, we compare the variational ground- stand for imaginary time. A first-order Trotter-Suzuki de- 48 state energy obtained by the numerical methods and discuss composition is used to efficiently represent the imaginary −τH the iPEPS results by studying the variational energy of com- time-evolution operator e . We also use the so-called full- 20,49 petitive ordered states and behavior of the spin correlation update scheme to truncate the bond dimension: at each length. In Sec. III B, we provide a systematic study of the step, imaginary time-evolution operator increases the bond di- nematic and magnetic order parameters and also discuss the mension of virtual bonds, so a truncation procedure is needed behavior of the gap by using DMRG.