preprint(January 15, 2021) Intriguing magnetism of the topological kagome magnet TbMn6Sn6 C. Mielke III,1, 2 Wenlong Ma,3 V. Pomjakushin,4 O. Zaharko,4 X. Liu,2 J.-X. Yin,5 S.S. Tsirkin,2 T.A. Cochran,5 M. Medarde,6 V. Por´ee,4 D. Das,1 C.N. Wang,1 J. Chang,2 T. Neupert,2 A. Amato,1 S. Jia,3, 7 M.Z. Hasan,5 H. Luetkens,1 and Z. Guguchia1, ∗ 1Laboratory for Muon Spin Spectroscopy, Paul Scherrer Institute, CH-5232 Villigen PSI, Switzerland 2Department of Physics, University of Z¨urich,Winterthurerstrasse 190, Zurich, Switzerland 3International Center for Quantum Materials, School of Physics, Peking University, Beijing, China 4Laboratory for Neutron Scattering and Imaging, Paul Scherrer Institut, CH-5232 Villigen PSI, Switzerland 5Laboratory for Topological Quantum Matter and Spectroscopy, Department of Physics, Princeton University, Princeton, New Jersey 08544, USA 6Laboratory for Multiscale Materials Experiments, Paul Scherrer Institut, CH-5232 Villigen PSI, Switzerland 7CAS Center for Excellence in Topological Quantum Computation, University of Chinese Academy of Science, Beijing, China Magnetic topological phases of quantum matter are an emerging frontier in physics and material science [1{10]. Along these lines, several kagome magnets have ap- peared as the most promising platforms. Here, we explore magnetic correlations in the transition-metal-based kagome magnet TbMn6Sn6 [11]. Our results show that the system exhibits an out-of-plane ferrimagnetic structure P 6=mm0m0 (comprised by Tb and Mn moments) with slow magnetic fluctuations in a wide temperature range. ∗ These fluctuations exhibit a slowing down below TC1 ' 120 K and a slightly modified quasi-static magnetic state is established below TC1 ' 20 K. A canted variation of the P 6=mm0m0 structure is possible, where all moments contribute to a net c-axis ferri- magnetic state which exhibits zero net in-plane components. Alternatively, a small incommensurate k-vector could arise below TC1. We further show that the tempera- ture evolution of the anomalous Hall conductivity (AHC) is strongly influenced by the low temperature magnetic crossover. More importantly, the here identified magnetic state seems to be responsible for the large quasi-linear magnetoresistance as well as for the appearance of quantum oscillations, which are related to the quantized Lan- dau fan structure featuring a spin-polarized Dirac dispersion with a large Chern gap. Therefore the exciting perspective of a magnetic system arises in which the topological response can be controlled, and thus explored, over a wide range of parameters. I. INTRODUCTION which originates in frustrated interplanar exchange in- teractions that trigger strong magnetic fluctuations, was discussed as the reason for the topological Hall effect. With distinguished symmetry and associated geomet- This suggests that detailed experimental studies to un- rical frustration, the kagome lattice can host peculiar derstand the magnetic fluctuations as well as the static states including flat bands [1], Dirac fermions [2, 3] and magnetic structure of these systems is necessary to re- spin liquid phases [4, 5]. Magnetic kagome materials are veal the coupling between relativistic electrons and the an ideal setting in which strongly correlated topologi- magnetic properties. cal electronic states may emerge [6{12]. In particular, transition-metal-based kagome magnets [1{6, 13{18] are In this so-called 166 materials, TbMn6Sn6 contains emerging as outstanding candidates for such states, as the heavy rare earth Tb crystallizes in a HfFe6Ge6-type they feature both large Berry curvature fields and un- structure (space group P 6/mmm) and is composed of usual magnetic tunability. As an example, we estab- hexagonal Tb layers containing Sn atoms and Mn kagome lished Co3Sn2S2 as a material that hosts frustrated mag- nets, stacked in the sequence -Mn-Tb-Mn-Mn-Tb-Mn- arXiv:2101.05763v1 [cond-mat.str-el] 14 Jan 2021 netism in the kagome lattice [6] and find that the volume- along the c-axis (Figure 2a) [19{25]. Structurally, com- wise magnetic competition drives the thermodynamic or pared with the Co3Sn2S2 system consisting of a Co3Sn quantum evolution of the intrinsic AHC, thereby tuning kagome layer and a Sn2 honeycomb layer, the RMn6Sn6 its topological state. A number of nontrivial magnetic system features a pristine Mn3 kagome layer (Fig. 2a), phases and a large topological Hall effect was also ob- as the Sn atoms are pushed away from the kagome layer served in another rare earth-transition metal based sys- by the chemical pressure from the R atoms. TbMn6Sn6 tem YMn6Sn6 [18]. A new nematic chirality mechanism, has two different magnetic sublattices: Tb- and Mn- sublattices. Previous neutron diffraction studies indi- cated that this compound exhibits in-plane ferrimagnetic ordering between Tb and Mn sublattices below the Curie ∗ Electronic address: [email protected] temperature TC = 423 K [19, 22], due to the strong Tb- 2 Mn exchange interaction. It was also reported that when ab-plane (Fig. 1b) along the direction of the magnetic the temperature drops to TC2 = 320 K [19, 22], a spin field. reorientation transition occurs and the easy magnetiza- tion direction changes from the ab-plane at high temper- atures to the c-axis at low temperatures. From the elec- We observe a large and sharp transition occurring tronic point of view, TbMn Sn exhibits Shubnikov-de 6 6 at T ' 320 K, corresponding to the spin reorien- Haas quantum oscillations with nontrivial Berry phases C2 tation transition noted in previous neutron diffraction at relatively low fields (from ∼ 7 T), a large AHC (0.14 studies [19, 22]. This marks the transition between e2/h per Mn kagome layer), arising from Berry curvature a high-temperature in-plane magnetic phase to a low- fields, and quasi-linear (/ H1:1) magnetoresistance (MR) temperature ferrimagnetic phase with magnetic moments likely resulting from linearly dispersive electrons [11, 16]. oriented out-of-plane, along the crystallographic c-axis. Moreover, TbMn Sn was found to demonstrate a bulk- 6 6 More importantly, when the magnetic field was applied boundary correspondence between the Chern gap and the along the c-axis, we observed a large reduction of zero- topological edge state, as well as Berry curvature field field cooled susceptibility χ below T ∗ ' 120 K correspondence of Chern-gapped Dirac fermions [11, 16]. ZFC C1 Thus, it is identified as a promising topological magnetic and settling into a negligibly small slightly diamagnetic system [11, 16]. Other than the identified high static plateau below TC1 ' 20 K. On the other hand, the field cooled susceptibility χFC shows a weak temperature de- magnetic ordering temperatures of TbMn6Sn6, micro- scopic studies of spin fluctuations and its tunability re- pendence across 120 K and down to the base-T . This main largely unexplored. gives rise to a much larger hysteresis in TbMn6Sn6 be- low 120 K than in the T -range of 120 K { 320 K. The Here we utilize high resolution muon-spin rotation plateau completely disappears with the application of (µSR), a very powerful local magnetic probe, in combina- higher magnetic fields, as it is not visible with the ap- tion with magnetization, powder and single crystal neu- plication of even a modest field of 70 mT. The reduction tron diffraction to systematically characterize the phase of χ is also less pronounced in 70 mT. The susceptibil- diagram, uncovering intriguing slow magnetic fluctua- ZFC ity, measured for a field applied in the ab-plane, is largely tions in TbMn Sn in a wide temperature range down 6 6 insensitive to the low-T transition (see Fig. 1b). A closer to 1.7 K. These fluctuations slow down below T ∗ ' 120 C1 view, shown in the inset of Fig. 1b, presents weak hys- K and freeze into a new quasi-static magnetic state be- teresis below T ∗ ' 120 K, suggesting a slightly differ- low T ' 20 K, which is suggested to be different and C1 C1 ent low temperature ferrimagnetic phase with moments more disordered than the ideal out-of-plane ferrimagnetic having a very weak component in the ab-plane. So, the state, observed at higher temperatures. T is considered C1 large difference between the FC and ZFC response is not as a magnetic crossover temperature signaling a slowing caused by the appearance of the in-plane structure but, down of magnetic fluctuations rather than being regarded rather, is consistent with the scenario that different out- as a true phase transition temperature. We also find of-plane ferrimagnetic domains tend to cancel out (anti- that the AHC does not follow the temperature evolu- align) after ZFC. If we FC the sample even in low fields, tion of the static Tb and Mn moments, but it is strongly then the domains align. This suggests that both mag- influenced by the observed low-temperature magnetic netic orders above and below 120 K are c-axis aligned, crossover. Moreover, the large linear MR and the quan- but slightly different from each other. This conclusion is tum oscillations, which are related to the quantized Lan- also substantiated by the field dependent measurements dau fan structure featuring a spin-polarized Dirac dis- of ZFC magnetization at T = 1.8 K, 250 K, and at 350 persion with a large Chern gap, appear below T ∗ and C1 K, measured for the field applied along the c-axis (Figure T , respectively. Thus, an intimate coupling between C1 1c) and along the ab-plane (Figure 1d). The critical fields the novel low-T magnetic state and the quantum-limit above which the magnetization plateau appears are sim- Chern gapped phase of TbMn Sn is concluded. 6 6 ilar for T = 1.8 K and 250 K. The only difference is that the data collected at 1.8 K show a plateau up to 12 mT, before increasing and then entering the linear regime, im- II.