Selected for a Viewpoint in Physics week ending PRL 103, 266801 (2009) PHYSICAL REVIEW LETTERS 31 DECEMBER 2009

Hexagonal Warping Effects in the Surface States of the Topological Insulator Bi2Te3

Liang Fu* Department of Physics, Harvard University, Cambridge, Massachusetts 02138, USA (Received 12 August 2009; revised manuscript received 21 October 2009; published 21 December 2009) A single two-dimensinoal Dirac fermion state has been recently observed on the surface of the topo- logical insulator Bi2Te3 by angle-resolved photoemission spectroscopy. We study the surface band struc- ture using k p theory and find an unconventional hexagonal warping term, which is the counterpart of cubic Dresselhaus spin-orbit coupling in rhombohedral structures. We show that this hexagonal warping term naturally explains the observed hexagonal snowflake Fermi surface. The strength of hexagonal warping is characterized by a single parameter, which is extracted from the size of the Fermi surface. We predict a number of testable signatures of hexagonal warping in spectroscopy experiments on Bi2Te3.We also explore the possibility of a spin-density wave due to strong nesting of the Fermi surface.

DOI: 10.1103/PhysRevLett.103.266801 PACS numbers: 73.20.r, 73.43.Cd, 75.10.b

Recently a new state of matter called a topological density wave (SDW) phase. We discuss various types of insulator has been observed in a number of materials [1– SDW order in a Landau-Ginzburg theory. 5]. A topological insulator has a time-reversal-invariant Bi2Te3 has a rhombohedral crystal structure with space band structure with nontrivial topological order, which group R3m . In the presence of a [111] surface, the sym- gives rise to gapless surface states bound to the sample metry of the crystal is reduced to C3v, which consists of a boundary [6–8]. The two-dimensional surface band has a threefold rotation C3 around the trigonal z axis and a mirror unique Fermi surface that encloses an odd number of Dirac operation M : x !x where x is in K direction. Two points in the surface Brillouin zone [6], which is prohibited surface bands are observed to touch at the origin of the in conventional materials by fermion doubling theorem [9]. surface Brillouin zone . The degeneracy is protected by Soon after the theoretical prediction [10], the semiconduct- time-reversal symmetry and the doublet jc ";#i forms a ing alloy BixSb1x was found to be a topological insulator Kramers pair. We choose a natural basis for the doublet having a Dirac surface band as well as other electron and according to total angular momentum J ¼ L þ S ¼1=2 i =3 2 hole pockets [1]. Subsequently, a family of materials Bi2X3 so that C3 is represented as e z . Since M ¼1 for ¼ Se 1 1 (X and Te) was found to be topological insulators spin 1=2 electron and MC3M ¼ C3 , the mirror opera- with a single Dirac-fermion surface state [3–5]. The ob- tion can be represented as M ¼ ix by defining the phase servation of an undoubled Dirac fermion is not only of of jc ";#i appropriately. The antiunitary time-reversal op- great conceptual interest but also paves the way for study- eration is represented by iyK (K is complex conjuga- ing unusual electromagnetic properties [10,11] and realiz- tion) and commutes with both M and C3. Here the ing topological quantum computation [12]. Therefore pseudospin i is proportional to the electron’s spin: hszi/ surface states of Bi2X3 are being intensively studied in hzi and hsx;yi/hx;yi. transport and spectroscopy experiments [13,14]. The Kramers doublet is split away from by spin-orbit In this work, we study the electronic properties of sur- interaction. We study the surface band structure near Bi Te face states in 2 3 using k p theory. Our motivation is using k p theory. To lowest order in k, the 2 2 effective to understand the shape of Fermi surface observed in recent angle-resolved photoemission spectroscopy (ARPES) ex- periments [4,5], reproduced in Fig. 1. By considering the crystal symmetry of Bi2Te3, we find an unconventional hexagonal warping term in the surface band structure, which is the counterpart of cubic Dresselhaus spin-orbit coupling in rhombohedral structures. This hexagonal warping term naturally explains the snowflake shape of the Fermi surface, and its magnitude is extracted from the size of the Fermi surface. We predict that hexagonal warp- ing of the Fermi surface should have important effects in FIG. 1 (color online). (i) Snowflakelike Fermi surface of the several spectroscopy experiments. Finally, we observe that surface states on 0.67% Sn-doped Bi2Te3 observed in ARPES. the Fermi surface of Bi2Te3 is nearly a hexagon with strong (ii) A set of constant energy contours at different energies. From nesting for an appropriate range of surface charge density. Y.L. Chen et al., Science 325, 178 (2009). Reprinted with This motivates us to explore theoretically a possible spin- permission from AAAS.

0031-9007=09=103(26)=266801(4) 266801-1 Ó 2009 The American Physical Society week ending PRL 103, 266801 (2009) PHYSICAL REVIEW LETTERS 31 DECEMBER 2009

Hamiltonian reads H0 ¼ vðkxy kyxÞ, which describes The hexagonal warping term Hw describes cubic spin- an isotropic 2D Dirac fermion. The form of H0 is strictly orbit coupling at the surface of rhombohedral crystal sys- fixed by symmetry. In particular, the Fermi velocity v in x tems, and, to the best of our knowledge, it has not been and y directions are equal because of the C3 symmetry. The reported before. It is instructive to compare Hw with the Fermi surface of H0 at any Fermi energy is a circle. well-studied trigonal warping in graphene [15]. Although However, the Fermi surface observed in ARPES, repro- graphene’s band structure also has Dirac points and its ~ duced in Fig. 1(i), is noncircular but snowflakelike: it has k p Hamiltonian HKðkÞ has C3v symmetry, the warping relatively sharp tips extending along six M directions and term in graphene is of a completely different form. This is curves inward in between. Moreover, as shown in Fig. 1(ii) because time-reversal operation acts differently for spin (we refer the reader to the original work [4] for better 1=2 ( ¼ iyK) and spinless fermions ( ¼ K). In gra- resolution), the shape of constant energy contour is phene time-reversal symmetry takes the latter form, and, ¼ energy-dependent, evolving from a snowflake at E together with inversion symmetry, leads to H ðk~Þ¼ 0:25 eV to a hexagon and then to a circle near the Dirac K H ðk~Þ ( ¼1 denote two sublattices), as opposed point. Throughout this Letter, energy is measured with x K x z to its partner Eq. (2)inBi2Te3. As a result, a different respect to the Dirac point. 2 2 trigonal warping term (k þ þ k) is symmetry- The observed anisotropic Fermi surface can only be þ allowed in graphene. explained by higher order terms in the k p Hamiltonian We now show that H naturally explains the observed Hðk~Þ that breaks the emerging Uð1Þ rotational symmetry of w energy-dependent shape of the Fermi surface in Bi2Te3. H0. The form of Hðk~Þ is highly constrained by crystal and ð ~Þ Using (4) we plot a set of constant energy contours ofpHffiffiffiffiffiffiffiffiffik time-reversal symmetry. Under the operation of C3 and M, for 0 Ec 7=6 E 0:69E , the edge of the hexagon curves inward so that Fermi surface ceases We then find that Hðk~Þ must take the following form up to to be purely convex. As E further increases, rounded tips third order in k~: start to develop at the vertices of the hexagon, which eventually become sharper, making the Fermi surface ð ~Þ¼ ð Þþ ð Þþ ð 3 þ 3 Þ snowflakelike. The evolution of the Fermi surface with H k E0 k vk kxy kyx 2 kþ k z; respect to energy matches well with the ARPES result (3) shown in Fig. 1. Moreover, it follows from (4) that the 2 where E0ðkÞ¼k =ð2m Þ generates particle-hole asymme- vertices of the hexagon—where the Fermi surface extends 2 outmost—always lie along M independent of the sign of try and the Dirac velocity vk ¼ vð1 þ k Þ contains a second-order correction. The last term in (3), which we , in agreement with ARPES data. call Hw, is most important. Unlike the other terms, Hw is only invariant under threefold rotation (as the Bi2Te3 crystal structure does) and therefore is solely responsible for the hexagonal distortion of the otherwise circular Fermi ~ surface. We note that HwðkÞ vanishes in mirror-symmetric z ~ direction M because is odd under mirror, and HwðkÞ reaches maximum along K. The surface band dispersion of Hðk~Þ is qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ð ~Þ¼ ð Þ 2 2 þ 2 6cos2ð3 Þ E k E0 k vkk k : (4)

Here E denote the energy of upper and lower band, and is the azimuth angle of momentum k~ with respect to the x ð ~Þ axis ( K). Although the Hamiltonian H is threefold invari- FIG. 2 (color online). (a) Constantpffiffiffiffiffiffiffiffiffi energy contour of H k . kx ! ant, the band structure is sixfold symmetric under and ky axis are in the unit of v=. (b) Constant energy contour þ 2=6 because of time-reversal symmetry. at E ¼ 1:2E is superimposed on the Fermi surface of Bi2Te3. 266801-2 week ending PRL 103, 266801 (2009) PHYSICAL REVIEW LETTERS 31 DECEMBER 2009

Comparing the set of Fermi surfaces in Fig. 2(a) with the of states (LDOS) around a nonmagnetic point defect in real Fermi surface in 0.67% Sn-doped Bi2Te3 (Fig. 1), we STM. The LDOS oscillation at a fixed energy decays find the Fermi surface at EF ¼ 1:2E is almost identical to algebraically as a function of distance away from the the one measured in ARPES, as shown by superimposing defect. In a normal 2D metal, the leading order (1=x) decay the two in Fig. 2(b). By fitting the theoretical value of of LDOS in a given direction x^ comes from scattering Fermi momentum along M (1:2=a) to the experimental between states at ‘‘stationary points’’ on the Fermi surface, one (0:11 A 1), we find a ¼ 10:9 A . Using the measured where the Fermi velocity is parallel to x^ [18]. For a convex Fermi velocity v ¼ 2:55 eV A, we obtain the magnitude constant energy contour below Ec as shown in Fig. 3(a), of the hexagonal warping term: ¼ 250 eV A 3. From only a single pair of stationary points exists at k~ and k~. that we find E ¼ 0:23 eV and EF ¼ 1:2E ¼ 0:28 eV However, since the two states at k~ and k~ here carry which agrees fairly well with the measured Fermi energy opposite spins, scattering between them is forbidden by 0.25 eV [shown in Fig. 1(ii)]. The quantitative agreement time-reversal symmetry—a fundamental property of sur- between theory and experiment suggests that the face states on a topological insulator. The LDOS oscilla- Hamiltonian (3) describes the surface band structure of tion then vanishes at leading order. Now consider a Bi Te 2 3 quite well in a wide energy window at least up to nonconvex constant energy contour above Ec. As shown 0.25 eV. As an independent check of the theory, we con- in Fig. 3(b), multiple pairs of stationary points exist. Since sider the nonlinear correction to surface band dispersion interpair scattering is still allowed, the LDOS oscillation near . Equation (4) predicts that the leading order correc- will be restored at leading order. Therefore, according to tion due to Hw starts at fifth order in k and is angle- the convexity of constant energy contour, two types of dependent: Friedel oscillation patterns should appear at different ranges of bias voltage. Eðk; Þ¼va4k5cos2ð3Þ=2: (5) In the last part of this work, we explore the possibility of Since the surface band dispersions along K and M a SDW phase on the surface of Bi2Te3. We note that the directions have been measured in ARPES [4,5], Eq. (5) Fermi surface is nearly a hexagon for 0:55E energy gap at the Dirac point. Consider an in-plane magnetic field Bk, which only couples to the spin k HZeeman ¼ gkB~k ~ . From (3) we find the Dirac point is FIG. 3 (color online). Illustration of scattering processes due to shifted away from to k~ gkz^ B~k=v. In addition to a point defect that causes the oscillation of LDOS. In a given ~ 3 direction x^ along M, the oscillation is dominated by scattering that, a mass term is generated at k : Mz ¼ðgkBkÞ 2 between stationary points marked by dots, where the Fermi sinð3’Þ =E (’ is the angle between B~k and K), which z velocity is parallel to x^. (a) A convex constant energy contour opens up an energy gap. When the Fermi energy is tuned, has a single pair of stationary points at k~ and k~. (b) A non- e.g., by doping [4,5], to lie within the gap, the insulating convex constant energy contour has three pairs of stationary state at the surface realizes quantum Hall effect without points. Intrapair scatterings in (a) and (b) are forbidden by time- Landau levels [17]. reversal symmetry. But interpair scatterings in (b), for example, Third, hexagonal warping of the Fermi surface has those between k2 and k3, are allowed. Therefore, the LDOS drastic effects on the Friedel oscillation of local density oscillation at leading order is absent in (a) but exists in (b). 266801-3 week ending PRL 103, 266801 (2009) PHYSICAL REVIEW LETTERS 31 DECEMBER 2009 follows, only one of i, say, 1, is nonzero. The resulting SDW X forms a one-dimensional stripe, which breaks C3 but is ¼ h y e i 0 j j¼j j¼ ik ckþQ i c~ k ; invariant under mirror symmetry. For u> , 1 2 i j j Xk 3 in the ordered phase, so that SDW forms a two- y ? ¼ i hc ðz^ e Þc~ i dimensional lattice. Each individual phase of i depends i kþQi i k (6) on the choice of origin. Only the global phase of 123 is Xk ¼ h y z i gauge invariant and is fixed by the sixth-order term of the iz i ckþQ ck ; 2 2 i form Cð123Þ þ C ð Þ in F. k 1 2 3 This work was initiated at University of Pennsylvania. y ¼ð y y Þ We thank Charlie Kane for inspiring discussions. We are where ck c"k;c#k are electron creation operators. For especially indebted to Bertrand Halperin for many insight- each Qi, we have chosen a local frame for in-plane spin components labeled by k and ? which are parallel and ful discussions and helpful comments on the manuscript. We thank Zahid Hasan and David Hsieh for numerous perpendicular to Qi, respectively. The order parameters thus defined transform nicely under the operations of ro- discussions on ARPES, as well as Aharon Kapitulnik, tation, mirror, time reversal, and translation: Anton Akhmerov, and especially Cenke Xu for useful conversations. This work was supported by the Harvard Society of Fellows and NSF Grant No. DMR-0605066. C3: i ! iþ1; Note added.—During the final stage of this work, we ! : i; i; learned that Alpichshev et al. [19] imaged with STM the (7) $ $ standing wave of surface states on Bi2Te3 near a line defect Mx: 1 1;2 3; (instead of a point defect considered in this work). The ! iQi d ¼k ? Td: i e i; ; ;z: LDOS oscillation was found to exist in the energy range with snowflakelike constant energy contour, but strongly We remark that because spin and momentum are locked by suppressed in the range with circular constant energy con- ð2Þ spin-orbit coupling, there is no SU symmetry for spin tour. This supports our explanation of the correlation be- alone. Thanks to the appropriate choice of order parame- tween LDOS oscillation and convexity of constant energy ters, symmetry operations (7) only act in the space of contour. ordering wave vectors labeled by i index. The Landau free energy F must be invariant under these symmetry operations. Only terms with even powers of i can exist because of time-reversal symmetry. At second *[email protected] order, we have [1] D. Hsieh et al., Nature (London) 452, 970 (2008); Science 1 X3 323, 919 (2009). ¼ F2 2 ii; (8) [2] A. Nishide arXiv:0902.2251. i¼1 [3] Y. Xia et al., Nature Phys. 5, 398 (2009). [4] Y.L. Chen et al., Science 325, 178 (2009). where the susceptibility matrix is real and symmetric [5] D. Hsieh et al., Nature (London) 460, 1101 (2009). because of mirror symmetry. is positive definite in the [6] L. Fu, C. L. Kane, and E. J. Mele, Phys. Rev. Lett. 98, normal state. When the temperature is lowered below Tc, 106803 (2007). one of the eigenvalues of first becomes negative, and [7] J. E. Moore and L. Balents, Phys. Rev. B 75, 121306(R) the surface undergoes a transition to a SDW. The spin (2007). configuration is then determined by the corresponding [8] R. Roy, arXiv:cond-mat/0607531. eigenvector v . For example, for a stripe SDW along the [9] H. Nielssen and N. Ninomiya, Phys. Lett. B 130, 389 (1983). ~ð Þ¼ð cosð Þ sinð Þ sinð ÞÞ x direction, S x; y vk Qx ;v? Qx ;vz Qx [10] L. Fu and C. L. Kane, Phys. Rev. B 76, 045302 (2007). with an appropriate choice of origin. [11] X. L. Qi, T. L. Hughes, and S. C. Zhang, Phys. Rev. B 78, The free energy (8) to second-order has an emerging 195424 (2008). Uð3Þ symmetry i ! Uijj. So single- and multiple-Q [12] L. Fu and C. L. Kane, Phys. Rev. Lett. 100, 096407 (2008). SDWs are degenerate. We now show that higher order [13] Y.S. Hor et al., Phys. Rev. B 79, 195208 (2009). terms in F break the Uð3Þ symmetry and pick out a [14] G. Zhang et al., Appl. Phys. Lett. 95, 053114 (2009). [15] R. Saito, G. Dresselhaus, and M. S. Dresselhaus, Phys. particular spatial ordering pattern.P For that purpose, it is 2 Rev. B 61, 2981 (2000). convenient to write ¼ v ; j j ¼ 1 and use as i i i i i [16] H. Zhang et al., Nature Phys. 5, 438 (2009). a new set of order parameters, which also transforms [17] F. D. M. Haldane, Phys. Rev. Lett. 61, 2015 (1988). according to (P7). At fourth order, we find an anisotropy [18] L. M. Roth, H. J. Zeiger, and T. A. Kaplan, Phys. Rev. 149, ¼ 3 j j4 term F4 u i¼1 i . The sign of u determines the 519 (1966). relative weight of i in the ordered phase. For u<0, [19] Z. Alpichshev et al., arXiv:0908.0371.

266801-4