Fermi Liquid Instabilities in the Spin Channel

SLAC-PUB-13919 Fermi liquid instabilities in the spin channel

Congjun Wu,1 Kai Sun,2 Eduardo Fradkin,2 and Shou-Cheng Zhang3 1Kavli Institute for Theoretical Physics, University of California, Santa Barbara, California 93106-4030 2Department of Physics, University of Illinois at Urbana-Champaign, Urbana, Illinois 61801-3080 3Department of Physics, McCullough Building, Stanford University, Stanford, California 94305-4045 (Dated: February 6, 2008) We study the Fermi surface instabilities of the Pomeranchuk type in the spin triplet channel with a high orbital partial waves (Fl (l > 0)). The ordered phases are classified into two classes, dubbed the α and β-phases by analogy to the superfluid 3He-A and B-phases. The Fermi surfaces in the α-phases exhibit spontaneous anisotropic distortions, while those in the β-phases remain circular or spherical with topologically non-trivial spin configurations in momentum space. In the α-phase, the Goldstone modes in the density channel exhibit anisotropic overdamping. The Goldstone modes in the spin channel have nearly isotropic underdamped dispersion relation at small propagating wavevectors. Due to the coupling to the Goldstone modes, the spin wave spectrum develops resonance peaks in both the α and β-phases, which can be detected in inelastic neutron scattering experiments. In the p-wave channel β-phase, a chiral ground state inhomogeneity is spontaneously generated due to a Lifshitz-like instability in the originally nonchiral systems. Possible experiments to detect these phases are discussed.

PACS numbers: 71.10.Ay, 71.10.Ca, 05.30.Fk

I. INTRODUCTION the electron fluid behaves, from the point of view of its symmetries and of their breaking, very much like an electronic analog of liquid crystal phases [5, 6]. The charge The Landau theory of the Fermi liquid is one of the nematic is the simplest example of such electronic liquid most successful theories of condensed matter physics [1, crystal phases, a concept introduced in Ref. [7] to de- 2]. It describes a stable phase of dense interacting scribe the complex phases of strongly correlated systems fermionic systems, a Fermi liquid (FL). Fermi liquid the- such as doped Mott insulators. The charge nematic phase ory is the foundation of our understanding of conven- has also been suggested to exist in the high T materi- tional, weakly correlated, metallic systems. Its central as- c als near the melting of the (smectic) stripe phases [7, 8], sumption is the existence of well-defined fermionic quasi- and in quantum Hall systems in nearly half-filled Landau particles, single particle fermionic excitations which exist levels [9, 10]. Experimentally, the charge nematic phase as long-lived states at very low energies, close enough to has been found in ultra-high mobility two-dimensional the Fermi surface. In the Landau theory, the interac- electron gases (2DEG) in AlAs-GaAs heterostructures tions among quasi-particles are captured by a few Lan- s,a and quantum wells in large magnetic fields, in nearly dau parameters F , where l denotes the orbital angular l half filled Landau levels for N 2 at very low tem- momentum partial wave channel, and s, a denote spin peratures [10, 11, 12, 13]. Strong≥ evidence for a charge singlet and triplet channels, respectively. Physical quan- nematic phase has been found quite recently near the tities, such as the spin susceptibility, and properties of metamagnetic transition of the ultra-clean samples of the collective excitations, such as the dispersion relation of bilayer ruthenate Sr Ru O [14, 15]. zero sound collective modes, acquire significant but finite 3 2 7 renormalizations due to the Landau interactions. In the Electronic liquid crystal phases can also be realized FL phase, except for these finite renormalizations, the as Pomeranchuk instabilities in the particle-hole channel effects of the interactions become negligible at asymp- with non-zero angular momentum. This point of view totically low energies. It has, however, long been known has been the focus of much recent work, both in con- that the stability of the FL requires that the Landau pa- tinuum system [4, 16, 17, 18, 19] as well as in lattice rameters cannot be too negative, F s,a > (2l + 1), a l − systems [20, 21, 22, 23, 24], in which case these quantum result first derived by Pomeranchuk [3]. The most famil- phase transition involve the spontaneous breaking of the iar of these Pomeranchuk instabilities are found in the point symmetry group of the underlying lattice. The 2D a s-wave channel: the Stoner ferromagnetism at F0 < 1 quantum nematic Fermi fluid phase is an instability in the and phase separation at F s < 1. − 0 − d-wave (l = 2) channel exhibiting a spontaneous ellipti- It has been realized quite recently that when these cal distortion of the Fermi surface [4]. In all cases, these bounds are violated in a channel with a non-vanishing instabilities typically result in anisotropic Fermi surface angular momentum, there is a ground state instability distortions, sometimes with a change of the topology of in the particle-hole, spin singlet, channel leading to a the Fermi surface. Nematic phases also occur in strong spontaneous distortion of the Fermi surface. This is a coupling regimes of strongly correlated systems such as quantum phase transition to a uniform but anisotropic the Emery model of the high temperature superconduc- liquid phase of the fermionic system [4]. In such a phase tors [25]. If lattice effects are ignored, the nematic state

SIMES, SLAC National Accelerator Center, 2575 Sand Hill Road, Menlo Park, CA 94309 Work supported in part by US Department of Energy contract DE-AC02-76SF00515. 2 has Goldstone modes can be viewed as the rotation of the which was first proposed by Leggett in the context of distorted Fermi surface, i.e., the soft Fermi surface fluctu- superfluid 3He-B phase [1]. In fact, it is a particle-hole ations. Within a random phase approximation (RPA) ap- channel analog of the superfluid 3He-B phase where the proach [4], confirmed by a non-perturbative high dimen- pairing gap function is isotropic over the Fermi surface. sional bosonization treatment [17], the Goldstone modes The phases with anisotropic Fermi surfaces distortions were shown to be overdamped in almost all the propagat- are the analog of the 3He-A phase where the gap func- ing directions, except along the high symmetry axes of tion is anisotropic. The two possibilities of keeping or the distorted Fermi surface. The Goldstone mode couples breaking the shape of Fermi surfaces are dubbed β and strongly to electrons, giving rise to a non-Fermi liquid be- α-phases respectively, by analogy with B and A super- havior throughout the nematic Fermi fluid phase [4, 26]: fluid phases in 3He systems. An important difference is, in perturbation theory, the imaginary part of the elec- however, that while in the A and B phases of superfluid 2 tron self-energy is found to be proportional to ω 3 on 3He all the Fermi surface is gapped, up to a set of mea- most of the Fermi surface, except along four “nodal” di- sure zero of nodal points in the A phase, in the β and rections, leading to the breakdown of the quasi-particle α-phases of spin triplet Pomeranchuk systems no gap in picture. Away from the quantum critical point, this ef- the fermionic spectrum ever develops. fect is suppressed by lattice effects but it is recovered at An important common feature of the α and β-phases quantum criticality [21] and at high temperatures (if the is the dynamical appearance of effective spin-orbit (SO) lattice pinning effects are weak). Beyond perturbation couplings, reflecting the fact that in these phases spin and theory [26] this effect leads to a form of ‘local quantum orbit degrees of freedom become entangled [37]. Con- criticality’. ventionally, in atomic physics, the SO coupling origi- Richer behaviors still can be found in Pomeranchuk nates from leading order relativistic corrections to the instabilities in the spin triplet and l> 0 angular momen- Schrödinger-Pauli equation. As such, the standard SO tum channels [4, 8, 27, 28, 29, 30, 31, 32, 33, 34]. Typ- effects in many-body systems have an inherently single- ically, these class of instabilities break both the spacial particle origin, and are unrelated to many-body correla- (orbital) and spin SU(2) rotation symmetries. A p-wave tion effects. The Pomeranchuk instabilities involving spin channel instability was studied by Hirsch [27, 28]. The we are discussing here thus provide a new mechanism Fermi surfaces of spin up and down components in such to generate effective SO couplings through phase tran- a state shift along opposite directions. The p-wave chan- sitions in a many-body non-relativistic systems. In the nel instability was also proposed by Varma as a candidate 2D β-phase, both Rashba and Dresselhaus-like SO cou- for the hidden order appearing in the heavy fermion com- plings can be generated. In the α-phase, the resulting SO pound URu2Si2 [31, 32] below 17 K. Gor’kov and Sokol coupling can be considered as a mixture of Rashba and studied the non-Néel spin orderings in itinerant systems, Dresselhaus with equal coupling strength. Such SO cou- and showed their relation to the Pomeranchuk instabil- pling systems could in principle be realized in 2D semi- ity [29]. Oganesyan and coworkers [4, 8] proposed the ex- conductor materials leading to interesting new effects. istence of a nematic-spin-nematic phase, a nematic state For instance, a hidden SU(2) symmetry in such systems in both real space and in the internal spin space, where was found to give rise to a long lived spin spiral excita- the two Fermi surfaces of up and down spins are sponta- tion with the characteristic wavevector relating the two neously distorted into two orthogonal ellipses. Podolsky Fermi surfaces together [38]. Recently, many proposals and Demler[35] considered a spin-nematic phase as aris- have been suggested to employ SO coupling in semicon- ing from the melting of a stripe phase; this phase can ductor materials to generate spin current through electric also be a nematic-spin-nematic if it retained a broken fields. The theoretical prediction of this “intrinsic spin rotational symmetry, as expected from the melting of a Hall effect” [39, 40] has stimulated tremendous research stripe (or smectic) phase[7, 36]. Kee and Kim [33] have activity both theoretical and experimental [41, 42]. Thus, suggested a state to explain the behavior of Sr3Ru2O7 Pomeranchuk instabilities in the spin channel may have which can be shown to be a lattice version of a partially a potential application to the field of spintronics. polarized nematic-spin-nematic state. A systematic description of the high partial-wave chan- Interestingly, the spin triplet Pomeranchuk instabili- nel Pomeranchuk instabilities involving spin is still lack- ties can also occur without breaking rotational invariance ing in the literature. In this paper, we investigate this in real space, and keep the symmetries of the undistorted problem for arbitrary orbital partial wave channels in two Fermi surface. Wu et al. [30] showed that a state of dimensions (2D), and for simplicity only in the p-wave this sort can exist in the p-wave channel, with the dis- channel in three dimensions (3D). We use a microscopic tortions affecting only the spin channel. In this state, model to construct a general Ginzburg-Landau (GL) free there are two Fermi surfaces with different volumes, as energy to describe these instabilities showing that the in ferromagnetic systems. Although both the spatial and structure of the α and β-phases are general for arbitrary spin rotation symmetries are broken, a combined spin- values of l. The α-phases exhibit anisotropic relative dis- orbit rotation keeps the system invariant, i.e., the total tortions for two Fermi surfaces as presented in previous angular momentum is conserved. The broken symme- publications. We also investigate the allowed topological try in such a state is the relative spin-orbit symmetry excitations (textures) of these phases and find that a half- 3 quantum vortex-like defect in real space, combined with rent can be induced when a charge current flows through spin-orbit distortions. The β-phases at l 2 also have the system. In Section XI, we discuss the possible exper- a vortex configuration in momentum space≥ with wind- imental evidence for Pomeranchuk instabilities involving ing numbers l which are equivalent to each other by a spins. We summarize the results of this paper at Section symmetry transformation.± XII. Details of our calculations are presented in two ap- We study the collective modes in critical regime and pendices. In Appendix A we specify our conventions for in the ordered phases at zero temperature. At the quan- Landau parameters and in Appendix B we give details of tum critical point, as in the cases previously studied, the the Goldstone modes for spin oscillations in the α-phase. theory has dynamic critical exponent z = 3 for all values of the orbital angular momentum l. In the anisotropic α-phase, the Goldstone modes can be classified into den- II. MODEL LANDAU HAMILTONIAN sity and spin channel modes, respectively. The density channel Goldstone mode exhibits anisotropic overdamp- We begin with the model Hamiltonian describing the a ing in almost all the propagating directions. In contrast, Pomeranchuk instability in the Fl (l 1) channel in 2D. the spin channel Goldstone modes show nearly isotropic Later on in the paper we will adapt this≥ scheme to discuss underdamped dispersion relation at small propagating the 3D case which is more complex. This model, and wavevectors. In the β-phase, the Goldstone modes are the related model for the spin-singlet sector of Ref. [4] relative spin-orbit rotations which have linear dispersion on which it is inspired, has the same structure as the relations at l 2, in contrast to the quadratic spin-wave effective Hamiltonian for the Landau theory of a FL. The dispersion in the≥ ferromagnet. Both the Goldstone modes corresponding order parameters can be defined through (spin channel) in the α-phases and the relative spin-orbit the matrix form as Goldstone modes in the β-phases couple to spin excita- µb † µ ˆ tions in the ordered phases. Thus the spin wave spec- Q (r) = ψ (r)σ gl,b( i )ψβ(r) (2.1) α αβ − ∇ tra develop characteristic resonance peaks observable in neutron scattering experiments, which are absent in the where the Greek indices µ denote the x,y,z directions normal phase. in the spin space, and in 2D the latin indices b = 1, 2 denote the two orbital components, and α, β = , label The p-wave channel Pomeranchuk instability involving ↑ ↓ spin is special because the Ginzburg-Landau (GL) free the two spin projections. (Hereafter, repeated indices are summed over.) In 2D, the operators g ig , which energy contains a cubic term of order parameters with a l,1 ± l,2 linear spatial derivative satisfying all the symmetry re- carry the azimuthal angular momentum quantum number L = l, are given by quirement. Such a term is not allowed in other chan- z ± nels with l = 1, including the ferromagnetic instability ˆ ˆ l ˆ ˆ l 6 gl,1( i ) igl,2( i ) = ( i) ( x i y) , (2.2) and the Pomeranchuk instabilities in the density chan- − ∇ ± − ∇ − ∇ ± ∇ nel. This term does not play an important role in the α- where the operator ˆ a is defined as ~ a/ . The 3D phase. But in the β-phase, it induces a chiral inhomoge- counterpart of these∇ expressions can be∇ written|∇| in terms neous ground state configuration leading to a Lifshitz-like of spherical harmonic functions. Thus, in 3D the latin instability in this originally nonchiral system. In other labels take 2l + 1 values. For the moment, and for sim- words, a Dzyaloshinskii-Moriya type interaction for the plicity, we will discuss first the 2D case. Goldstone modes is generated in the β-phase. The effect In momentum space (i.e. a Fourier transform) we can bears some similarity to the helimagnet [43] and chiral ~ liquid crystal [6] except that the parity is explicitly bro- write the operators of Eq. (2.2) in the form gl,1(k) = ~ ken there but not here. The spiral pattern of the ground cos lθk and gl,2(k) = sin lθk, where θk is the azimuthal state order parameter is determined. angle of ~k in the 2D plane. In momentum space, Qµb(~q) The paper is organized as follows: In Section II, we is defined as construct the model Hamiltonian for the Pomeranchuk ~q ~q instabilities with spin. In Section III, we present the Qµb(~q)= ψ† (~k + ) σµ g (~k) ψ (~k ). (2.3) α 2 αβ l,b β − 2 Ginzburg-Landau free energy analysis to determine the ~k allowed ground states. In Section IV, we discuss the re- X sults of the mean field theory. In Section V we discuss It satisfies Qµb( ~q)= Qµ,b,∗(~q), thus Qµb(~r) is real. the topology of the broken symmetry states and classify We generalize− the Hamiltonian studied in Ref. [4, 30] a the topological defects. In Section VI, we calculate the to the Fl channel with arbitrary values of l as collective modes at the critical point at zero temperature. In Section VII and Section VIII, we investigate H = d2~rψ† (~r)(ǫ(~ ) µ)ψ (~r) the Goldstone modes in the α and β-phases respectively, α ∇ − α Z and study the spontaneous Lifshitz instability in the β- 1 + dd~rdd~r′ f a(~r ~r′) Qˆµb(~r)Qˆµb(~r′), phase at l = 1. In Section IX, we present the magnetic 2 l − field effects. In Section X, we show that in the α and Z Xµb β-phases in the quadrupolar channel (l = 2), a spin cur- (2.4) 4

a where µ is the chemical potential. For later convenience, Taking into account that f1 1/N(0) around the we include the non-linear momentum dependence in the transition point, we introduce| a dimensionless| ∼ parameter single particle spectrum up to the cubic level as λ to denote the above criterion as

~ 2 κk2 ǫ(k)= vF ∆k(1 + a(∆k/kF )+ b(∆k/kF ) + . . .) (2.5) λ = F 1. (2.11) N(0) ≫ with ∆k = k kF . Here vF and kF are the Fermi velocity and the magnitude− of the Fermi wave vector in the FL. Finally, notice that, in the p-wave channel, the Hamilto- The Fourier transform of the Landau interaction func- nian Eq. (2.10) can be formally represented through an tion f(r) is SU(2) non-Abelian gauge field minimally-coupled to the fermions a i~q~r a fl f(~q)= d~re fl (r)= , (2.6) 1 1+ κ f a q2 H = d2~r ψ†(~r)( i~ a mAµ(~r)σµ)2ψ(~r) Z | l | mf 2m − ∇ − a Zm and the dimensionless Landau parameters are defined as ψ†(~r)ψ(~r)Aµ(~r)Aµ(~r). (2.12) − 2 a a F a = N(0)f a(q = 0) (2.7) l l where the gauge field is defined as with N(0) the density of states at the Fermi energy. This µ µ µa µ Hamiltonian possesses the symmetry of the direct prod- Aa (~r)σ = n (~r)σ . (2.13) uct of SOL(2) SOS(3) in the orbit and spin channels. ⊗ a Notice, however, that this is an approximate effective lo- The LP instability occurs at Fl < 2 at l 1 in two dimensions. For the general values of −l, we represent≥ the cal gauge invariance which only holds for a theory with a order parameter by a 3 2 matrix linear dispersion relation, and that it is manifestly bro- × ken by non-linear corrections, such as the quadratic term d2~k of Eq. (2.12), and the cubic terms included in the dis- nµ,b = f a ψ† (k)σµ g (~k)ψ (k) . (2.8) ~ | 1 | (2π)2 h α αβ l,b β i persion ǫ(k) of Eq. (2.5). Z It is more convenient to represent each column of the matrix form nµ,b (b =1, 2) as a 3-vector in spin space as III. GINZBURG-LANDAU FREE ENERGY

µ,1 µ,2 ~n1 = n , ~n2 = n . (2.9) A. The 2D systems

For l = 1, ~n1,2 are just the spin currents along the x, y In order to analyze the possible ground state conﬁgu- directions respectively. When l 2, ~n1,2 denote the spin multipole components at the level≥ l on the Fermi surface. ration discussed in Section I, we construct the G-L free energy in 2D in the arbitrary l-wave channel. The sym- ~n1 i~n2 carry the orbital angular momenta Lz = l ± ± metry constraint to the G-L free energy is as follows. Un- respectively. In other words, ~n1,2 are the counterpart of the spin-moment in the l-th partial wave channel in der time-reversal (TR) and parity transformations, ~n1,2 momentum space. transform, respectively, as The mean ﬁeld Hamiltonian, i.e. for a state with a T~n T −1 = ( )l+1~n , P~n P −1 = ( )l~n . (3.1) uniform order parameter, can be decoupled as 1,2 − 1,2 1,2 − 1,2

d2~k Under the SOS(3) rotation Rµν in the spin channel, ~n1,2 H = ψ† (~k) ǫ(~k) µ ~n cos(lθ ) transform as MF (2π)2 α − − 1 k Z n 2 2 ~ n1 + n2 nµ,1 Rµν nν,1, nµ,2 Rµν nν,2, (3.2) + ~n2 sin(lθk) ~σ ψβ(k)+ | | a| | . → → · 2 fl o | | (2.10) On the other hand, under a uniform rotation by an angle θ about the z-axis, in the orbital channel, the order This mean field theory is valid when the interaction range parameters ~na transform as ξ κ f a is much larger than the inter-particle dis- ≈ | l | ~n cos(lθ) ~n + sin(lθ) ~n , tance d 1/kF . The actual validity of mean field theory 1 1 2 p ≈ → at quantum criticality requires an analysis of the effects ~n2 sin(lθ) ~n1 + cos(lθ) ~n2. (3.3) of quantum fluctuations which are not included in mean → − field theory [44, 45]. In this theory, just as the case of Thus, the order parameter fields ~n1 and ~n2 are invariant Ref. [4], the dynamic critical exponent turns out to be under spatial rotations by 2π/l, and change sign under z = 3 and mean field theory appears to hold even at a rotation by π/l. In the α-phase this change change be quantum criticality. compensated by flipping the spins. 5

In order to maintain the SO (2) SO (3) symmetry, be written as L ⊗ S up to quartic terms in the order parameter fields ~na, the 2 uniform part of the GL free energy has the form F (n) = γ (∂ δ igAλ(T λ) ) nνb grad 1 a µν − a µν n2 λa νb 2 o T v2 T 2 v2 T 2 + γ1g (ǫλµν n n ) F (n) = rtr(n n) + (v1 + )[tr(n n)] tr[(n n) ] 2 − 2 2 = γ1(∂a~nb + g ~na ~nb) = r( ~n 2 + ~n 2)+ v ( ~n 2 + ~n 2)2 × 1 2 1 1 2 ab | | | 2 | | | | | X n + v2 ~n1 ~n2 . (3.4) 2 2 | × | γ1g ~na ~nb , (3.8) − | × | The coefficients r, v , v will be presented in Eq. (4.12) o 1 2 with g = γ /(2γ ). by evaluating the ground state energy of the mean field 2 1 Hamiltonians Eq. (4.1) and Eq. (4.7). The Pomeranchuk instability occurs at r < 0, i.e., F a < 2 (l 1). Fur- l − ≥ B. The 3D systems thermore, for v2 > 0, the ground state is the α-phase which favors ~n1 ~n2, while leaving the ratio of ~n1 / ~n2 k | | | | In 3D, the order parameter in the F a channel Pomer- arbitrary. On the other hand, for v2 < 0 we find a β- l anchuk instabilities can be similarly represented by a phase, which favors ~n1 ~n2 and ~n1 = ~n2 . ⊥ | | | | 3 (2l + 1) matrix. Here we only consider the simplest The gradient terms are more subtle. We present the case× of the p-wave channel instability (l = 1), which has gradient terms of the GL free energy for l = 1 as follows been studied in Ref. [4, 27, 30, 32] under different con- a µ,i T µa νb λb texts. In the F1 channel, the order parameter n is a Fgrad(n) = γ1tr[∂an ∂an]+ γ2ǫµνλn n ∂an 3 3 real matrix defined as × = γ (∂ ~n ∂ ~n )+ γ (∂ ~n ∂ ~n ) 1 a b a b 2 x 2 y 1 3~ · − µ,b a d k † µ b n n = f1 ψα(k)σ k ψβ(k) . (3.9) (~n ~n ) . (3.5) | | (2π)3 h αβ i · 1 × 2 Z o The difference between nµ,i and the triplet p-wave pairing For simplicity, as in Ref. [4], we have neglected the dif- order parameter in the 3He system [1, 46] is that the ference between two Frank constants and only present former is defined in the particle-hole channel, and thus is one stiffness coefficient γ1. (This approximation is accu- real. By contrast, the latter one is defined in the particle- rate only near the Pomeranchuk quantum critical point.) particle channel and is complex. Each column of the µ,b More importantly, because n is odd under parity trans- matrix form nµ,b(b = x,y,z) can be viewed as a 3-vectors formation for l = 1, a new γ2 term appears, which is in the spin space as of cubic order in the order parameter field nµ,b, and it µ,1 µ,2 µ,3 is linear in derivatives. This term is allowed by all the ~n1 = n , ~n2 = n , ~n3 = n , (3.10) symmetry requirements, including time reversal, parity, and rotation symmetries. This term has no important which represents the spin current in the x, y and z direc- effects in the disordered phase and in the α-phase, but tions respectively. The G-L free energy in Ref. [30] can it leads to a Lifshitz-like inhomogeneous ground state be reorganized as with spontaneous chirality in the β-phase in which par- ′ ′ ity is spontaneously broken. We will discuss this effect ′ T ′ v2 T 2 v2 T 2 F (n) = r tr(n n) + (v1 + )[tr(n n)] tr[(n n) ] in detail in Section VIII. The coefficient of γ1,2 will be 2 − 2 ′ 2 2 2 ′ 2 presented in Eq. (8.19). Similarly, for all the odd values = r ( ~n1 + ~n2 + ~n3 )+ v1( ~n1 of l, we can write a real cubic γ term satisfying all the | | | | | | | | 2 + ~n 2 + ~n 2)2 + v′ ~n ~n 2 symmetry constraints as | 2| | 3| 2 | 1 × 2| 2 n 2 µa νb l λb + ~n2 ~n3 + ~n3 ~n1 , (3.11) γ2ǫµνλn n (i) ga( ˆ )n . (3.6) | × | | × | ∇ o where the coefficients r′, v′ will be presented in Eq. However, this term corresponds to high order corrections, 1,2 (4.19). Similarly, the α-phase appears at v2 > 0 which and is negligible (irrelevant) for l 3. favors that ~n ~n ~n , and leaves their ratios arbi- ≥ 1 k 2 k 3 Similarly to the approximate gauge symmetry for the trary. The β-phase appears at v2 < 0 which favors that fermions in Eq. (2.12), the γ2 term can also be repro- vectors ~n1,2,3 are perpendicular to each other with equal duced by a non-Abelian gauge potential defined as amplitudes ~n1 = ~n2 = ~n3 . Similarly,| we| present| | the| gradient| terms in the G-L free λ λ λa iAa (x)(T )µν = ǫλµν n (x), (3.7) energy as

′ T ′ µi νj λi λ Fgrad(n) = γ tr[∂an ∂an]+ γ ǫµνλn n ∂j n , where Tµν = iǫλµν is the generator of the SU(2) gauge 1 2 group in the− vector representation. Then Eq. (3.5) can (3.12) 6

δk↑ ~s The value ofn ¯ can be obtained by solving the self- consistent equation in the α-phase

n¯ d~k = n (ξ (~k)) n (ξ (~k)) cos(lθ ). f a(0) (2π)2 { f ↑ − f ↓ } k | l | Z (4.3)

where nf (ξ↑(k)) and nf (ξ↓(k)) are the Fermi functions ~s for the up and down electrons respectively. The dis- δk↓ tortions of the Fermi surfaces of the spin up and spin l = 1 α-phase l = 2 down bands are given by an angle-dependent part of their Fermi wave vectors:

a a δk (θ) 2a 1 a 2a2 FIG. 1: The α-phases in the F1 and F2 channels. The Fermi F ↑,↓ = − x2 (x + − x3)cos lθ surfaces exhibit the p and d-wave distortions, respectively. kF 4 ± 2 ax2 cos2 lθ (2a2 b)x3 cos3 lθ + O(x4), − ± − ′ (4.4) where the coefficient γ1,2 will be presented at Eq. (8.36). Again, we neglect the difference among three Frank con- ′ where we introduced the dimensionless parameter x = stants. The γ2 term can be represented as n/¯ (vF kF ), where vF and kF are the Fermi velocity and the magnitude of the Fermi wave vector in the FL phase. γ′ (∂ ~n ∂ ~n ) (~n ~n ) 2 x 2 − y 1 · 1 × 2 This solution holds for small distortions and it is accurate n +(∂y~n3 ∂z~n2) (~n2 ~n3) only close to the quantum phase transition. By inspec- − · × tion we see that in the α-phases the total spin polariza- +(∂ ~n ∂ ~n ) (~n ~n ) . (3.13) z 1 − x 3 · 3 × 1 tion vanishes. More importantly, under a rotation by π/l, o the charge and spin components of the order parameter It can also be represented in terms of the non-Abelian both change sign, i.e. a rotation by π/l is equivalent to gauge potential as in Eq. (3.8). a reversal of the spin polarization. The single particle fermion Green function in the α- a phase at wave vector ~k and Matsubara frequency ωn is IV. MEAN FIELD PHASES IN THE Fl CHANNEL 1 1+ σz 1 σz G(~k,iωn)= + − . (4.5) 2 iωn ξ↑(~k) iωn ξ↓(~k) In this section, we discuss the solution to the mean field n − − o Hamiltonian Eq. (2.10), for the ordered α and β-phases in both 2D and 3D. B. The2D β-phases

The β-phase appears for v < 0, which favors ~n ~n A. The2D α-phases 2 1 ⊥ 2 and n1 = n2 . Like the case of ferromagnetism, the Fermi| surfaces| | | split into two parts with different vol- The α-phase is characterized by an anisotropic distor- umes, while each one still keeps the round shape undis- tion of the Fermi surfaces of up and down spins. In torted. However, an important difference exists between this phase s is as a good quantum number. It is a z the β-phase at l 1 and the ferromagnetic phase. In straightforward generalization of the nematic Fermi liq- the ferromagnet, the≥ spin is polarized along a fixed uni- uid for the case of the spin channel [4]. For example, form direction, which gives rise to a net spin moment. the Fermi surface structures at l = 1, 2 are depicted in On the other hand, in the β-phase with orbital angu- Fig. 1. The quadrupolar, l = 2, case is the nematic- lar momentum l 1, the spin winds around the Fermi spin-nematic phase [4, 8]. Without loss of generality, we surface exhibiting≥ a vortex-like structure in momentum choose ~n =n ¯ zˆ, and n = 0. The mean field Hamilto- 1 2 space. Consequently in the β-phase the net spin moment nian H for the α-phase| | becomes α is zero, just as it is in the α-phase. (Naturally, partially d2k polarized versions of the α and β-phases are possible but H = ψ†(~k)[ǫ(~k) µ n¯ cos(lθ )σ )]ψ(~k). α,l (2π)2 − − k z will not be discussed here.) In other words, it is a spin Z nematic, high partial wave channel generalization of fer- (4.1) a romagnetism. In the previously studied cases of the F1 The dispersion relations for the spin up and down elec- channel in Ref. [30], it was shown that in this phase trons become effective Rashba and Dresselhaus terms are dynamically generated in single-particle Hamiltonians. The ground ξ (~k) = ǫ(k) µ n¯ cos(lθ ), (4.2) state spin configuration exhibits, in momentum space, a ↑,↓ − ∓ k 7

δk↑ δk↑

δk↓ ~s ~s δk↓ ~s

(a) w = 2 (b) w = −2

a FIG. 3: The β-phases in the F2 channel. Spin conﬁgurations (a) Rashba (w = 1) exhibit the vortex structure with winding number w = ±2. These two conﬁgurations can be transformed to each other by performing a rotation around the x-axis with the angle of π.

~s ~n2

~n1

(b) Dresselhaus (w = −1) FIG. 4: The spin conﬁgurations on both Fermi surfaces in the a 2 β-phase (Fl channel) map to a large circle on an S sphere a FIG. 2: The β-phases in the F1 channel. Spin conﬁgurations with the winding number l. exhibit the vortex structures in the momentum space with winding number w = ±1, which correspond to Rashba and Dresselhaus SO coupling respectively. by solving the self-consistent equation

2~ 2¯n d k ~ ~ vortex structure with winding number w = 1 depicted a = 2 nf (ξ↑(k)) nf (ξ↓(k) , (4.8) ± fl (0) (2π) − in Fig. 2. | | Z Here we generalize the vortex picture in momentum a a where the single particle spectra read space in the F1 channel to a general Fl channel. We assume n = n =n ¯. Without loss of generality, we | 1| | 2| ξ (~k)= ǫ(~k) µ n,¯ (4.9) can always perform an SO(3) rotation in spin space to ↑,↓ − ± set ~n xˆ, and ~n yˆ. Then, much as in the B phase of 1 2 which is clearly invariant under spacial rotations. The 3He, thek mean ﬁeldk Hamiltonian H for the β-phase in β,l Fermi surface splits into two parts with angular momentum channel l can be expressed through a d-vector, deﬁned by δk x2 F,↑,↓ = x (a b)x3 + O(x4). (4.10) kF ± − 2 ± − d~(~k) = (cos(lθk), sin(lθk), 0) , (4.6) as follows The single particle Green function in the β-phase reads 2 ˆ ˆ d k † ~ ~ ~ ~ ~ 1 1+ ~σ d 1 ~σ d Hβ,l = 2 ψ (k) ǫ(k) µ n¯d(θk) ~σ) ψ(k), Gl(k,iωn)= · + − · , (4.11) (2π) − − · 2 iωn ξ↑(~k) iωn ξ↓(~k) Z n − − o h i (4.7) where ~σ d~(kˆ) is the l-th order helicity operator. Each · where d~(θk) is the spin quantization axis for single parti- Fermi surface is characterized by the eigenvalues 1 of ± cle state at ~k. The saddle point value ofn ¯ can be obtained the helicity operators ~σ d~(kˆ). · 8

From the mean field theory of α and β-phases, we can Now we discuss the general configuration of the d- a calculate the coefficients of the G-L theory, Eq.(3.4), as vector in the β-phase in the Fl channel. ~n1 and ~n2 can be any two orthogonal unit vectors on the S2 sphere. The N(0) 1 1 2 r = ( ), plane spanned by ~n1,2 intersects the S sphere at any 2 F a − 2 large circle as depicted in Fig. 4, which can always be | l | N(0) N ′(0) N ′′(0) obtained by performing a suitable SO(3) rotation from v = [ ]2 1 32 N(0) − 2N(0) the large circle in the xy plane. The spin configuration around the Fermi surface maps to this large circle with n N(0) o 2 the winding number of l. Furthermore, the configuration = (1 a 2a +3b) 2 2 , − − 32vF kF of winding number l are equivalent to each other up to N ′′(0) N(0) rotation of π around± a diameter of the large circle. For v = = ( a +2a2 b) , (4.12) 2 2 2 example, the case of w = 2 is depicted in Fig. 3 B, 48 − − 8vF kF which can be obtained from− that of w = 2 by performing where v1,2 do not depend on the value of l at the mean such a rotation around thex ˆ-axis. Similarly, with the ′ ′′ field level. N (0) and N (0) are the first and second SOS(3) symmetry in the spin space, the configurations order derivatives of density of states at the Fermi energy with w = l are topologically equivalent to each other. ± Ef respectively. They are defined as However, if the SOS(3) symmetry is reduced to SOS(2) because of the existence of an explicit easy plane mag- dN N(0) ′ netic anisotropy (which is an effect of SO interactions at N (0) = ǫ=Ef = (1 2a) , dǫ | − vf kf the single particle level), or by an external magnetic field 2 ′′ d N 2 N(0) B~ , then the two configurations with w = l belong to N (0) = 2 ǫ=Ef = 6( a +2a b) 2 2 . (4.13) ± dǫ | − − vf kf two distinct topological sectors. Both of them only depend on the non-linear dispersion relation up to the cubic order as kept in Eq. 2.5. C. The 3D instabilities of the p-wave spin triplet It is worth to stress that the coefficients of Eq. (4.12) channel were calculated (within this mean field theory) at fixed density. Similar coefficients were obtained in the spinless The mean field theory for the Pomeranchuk instability system analyzed in Ref. [4] at fixed chemical potential. in the F a channel has been studied in Ref. [30]. To The is a subtle difference between these two settings in 1 make the paper self-contained, here we summarize the the behavior of the quartic terms. At fixed chemical po- main results. tential the sign of b, the coefficient of the cubic term in the In the α-phase, taking the special case nµa =nδ ¯ δ , free fermion dispersion relation, is crucial for the nematic µz az the mean field Hamiltonian reads phase to be stable. However, as can be seen in Eq.(4.12), the sign of the coefficient of the quartic term v1 is deter- 3 d k † mined by several effects: the coefficients a and b, and that H2D,α = ψ (~k) ǫ(~k) µ nσ¯ z cos θk) ψ(~k), (2π)3 − − of an extra (additive) contribution which originates from Z h i the curvature of the Fermi surface and hence scales as (4.15) 2 N(0)/kF . It has been noted in Refs. [23, 47, 48] that the nematic instability for lattice systems may be a continu- where θ is the angle between ~k and z-axis. The Fermi ous quantum phase transition or a first order transition, surfaces for the two spin components are distorted in an in which case it involves a change in the topology of the opposite way as Fermi surface. As shown above, this dichotomy is the re- ∆k (θ) 1 2 sult of the interplay of the single particle dispersion and F ↑,↓ = (1 a)x2 [x + a(1 a)3x3]cos θ effects due to the curvature of the Fermi surface. The kF 3 − ± 3 − same considerations apply to the coefficients that we will ax2 cos2 θ (2a2 b)x3 cos3 θ + O(x4) − ± − present in the following subsection. (4.16) The ground state spin configuration in the β-phase exhibits a vortex structure with winding number w = l In the β-phase, rotational symmetry is preserved and in momentum space. The case of w = 2 is depicted in a SO interaction is dynamically generated. With the Fig. 3 A. Interestingly, in the case of l = 3, after setting µa ansatz n =nδ ¯ µa, the MF Hamiltonian reduces to d~ = (cos(3θ + π/2), sin(3θ + π/2), 0), the effective single particle Hamiltonian becomes H = ψ†(k)(ǫ(k) µ n~σ¯ kˆ)ψ(k). (4.17) 3D,β − − · H (k)= ǫ(k)+¯n [ sin(3θ )σ + cos(3θ )σ ] . (4.14) k MF − k x k y X This single particle Hamiltonian has the same form as is The single particle states can be classified according to that of the heavy hole band of the 2-dimensional n-doped the eigenvalues 1 of the helicity operator ~σ kˆ, with GaAs system [49, 50], which is results from SO coupling. dispersion relations± ξB(k) = ǫ(k) µ n¯. The· Fermi ↑,↓ − ± 9 surfaces split into two parts, but still keep the round z shape for two helicity bands with x y ∆k F ↑,↓ = x x2 (2a b)x3 + O(x4). (4.18) kF ± − ± −

The β-phase is essentially isotropic. The orbital angular momentum L~ and spin S~ are no longer separately conserved, but the total angular momentum J~ = L~ + S~ =0 remains conserved instead. For the general case of nµa = nD¯ µa, it is equivalent to a redefinition of spin operators ′ as Sµ = Sν Dνaδaµ, thus Fermi surface distortions remain isotropic and J~′ = L~ + S~′ is conserved. From the above mean field theory, we can calculate the FIG. 5: The half-quantum vortex with the combined spin- orbit distortion for the α-phase at l = 1. The triad denotes coefficients in Eq. (3.11) as the direction in the spin space. N(0) 1 1 r′ = ( ), 2 F a − 3 A. Topology of the α-phases | 1 | N(0) 1 N ′(0) N ′′(0) v′ = [ ]2 1 24 3 N(0) − 5N(0) 1. 2D α-phases n o N(0) 2 µa = 2 2 (7 2a 8a +9b), For the 2D α-phases, for which we can set n = 180vF kF − − h i nδ¯ µzδa1, the system is invariant under SOS(2) rotation ′ N2(0) N(0) 2 in the spin channel, the Zl rotation with the angle of 2π/l v2 = = 2 2 (1 6a +6a 3b), 90 45vF kF − − in the orbital channel, and a Z2 rotation with the angle of (4.19) π around the x-axis in the spin channel combined with an orbital rotation at the angle of π/l. Thus the Goldstone where manifold is

dN 2 2a [SOL(2) SOS(3)]/[SOS(2) ⋉ Z2 Zl] N ′(0) = = − N(0), ⊗ ⊗ dǫ |ǫ=Ef v k = [SO (2)/Z ] [S2/Z ], (5.1) f f L l ⊗ 2 2 2 ′′ d N 2(1 6a +6a 3b) N (0) = 2 ǫ=Ef = − 2 2 − N(0). giving rise to three Goldstone modes: one describes the dǫ | vf kf oscillation of the Fermi surface, the other two describe (4.20) the spin precession. Their dispersion relation will be calculated in Section VII A. Once again, the caveats of the previous subsection on the Due to the Z2 structure of the combined spin-orbit ro- sign of the coefficients of the quartic terms apply here too. tation, the vortices in the 2D-α-phase can be divided into two classes. The first class is the 1/l-vortices purely in the orbital channel without distortions in the spin channel. This class of vortices have the same structure as that V. GOLDSTONE MANIFOLDS AND in the Pomeranchuk instabilities in the density channel TOPOLOGICAL DEFECTS dubbed the integer quantum vortex. On the other hand, another class of vortices as combined spin-orbital defects In this Section we will discuss the topology of the bro- exist. This class of vortices bears a similar structure to ken symmetry α and β-phases, and their associated Gold- that of the half-quantum vortex (HQV) in a superfluid stone manifolds in 2D and 3D. We also discuss and clas- with internal spin degrees of freedom [51, 52, 53]. An sify their topological defects. We should warn the reader example vortex at l = 1 of this class is depicted in Fig. that the analysis we present here is based only on the 5 where the π-disclination in the orbit channel is offset static properties of the broken symmetry phases and ig- by the rotation of π around the x-axis in the spin chan- nores potentially important physical effects due to the nel. To describe the vortex configuration for the case of fact that in these systems the fermions remain gapless the p wave channel, we set up a local reference frame in (although quite anomalous). In contrast, in anisotropic spin space, and assume that the electron spin is either superconductors the fermion spectrum is gapped (up to parallel or anti-parallel to the z-axis of this frame, at a possibly a set of measure zero of nodal points of the Fermi point ~x in real space. Let φ = 0 be angular polar coor- surface). The physics of these effects will not be discussed dinate of ~x with respect to the core of the vortex. As we here. trace a path in real space around the vortex, the frame in 10 spin space rotates so that the spin flips its direction from Lz + σz/2 is still conserved. Generally speaking, the 2D up to down (or vice versa) as we rotate by an angle of β-phases with the momentum space winding number w π. (In the d-wave channel the vortex involves a rotation is invariant under the combined rotation generated by by π/2.) Such behavior is a condensed matter example Lz + wσz/2. The corresponding Goldstone manifold is of the Alice-string behavior in the high energy physics [54, 55]. Another interesting behavior of HQV is that [SOL(2) SOS (3)]/SOL+S(2) = SO(3). (5.3) a pair of half-quantum vortex and anti-vortex can carry ⊗ spin quantum number. This is an global example of the Three Goldstone modes exist as relative spin-orbit ro- Cheshire charge in the gauge theory [51, 53]. The elec- tations around x, y and z-axes on the mean field ground tron can exchange spin with the Cheshire charged HQV state. Similarly, for the 3D β-phase with l = 1, the Gold- pairs when it passes in between the HQV pairs. stone mode manifold is Due to the SO(2) SO (2) symmetry in the Hamil- L × S tonian, the fluctuations in the orbital channel are less se- [SOL(3) SOS(3)]/SO(3)L+S = SO(3), (5.4) vere than those in the spin channel. In the ground state, ⊗ the spin stiffness should be softer than that in orbital with three branches of Goldstone modes. The dispersion channel. As a result, an integer-valued vortex should relation of these Goldstone modes will be calculated in be energetically favorable to fractionalize into a pair of Section VIII. The vortex-like point defect in 2D and the HQV. However, at finite temperatures, in the absence line defect in 3D are determined by the fundamental ho- of magnetic anisotropy (an effect that ultimately is due motopy group π1(SO(3)) = Z2. On the other hand, the to spin orbit effects at the atomic level) the spin chan- second homotopy group π2(SO(3)) = 0, thus, as usual, nel is disordered with exponentially decaying correlation no topologically stable point defect exists in 3D. functions, and thus without long range order in the spin channel. However, the orbital channel still exhibits the Kosterlitz-Thouless behavior at 2D where the low energy VI. THE RPA ANALYSIS IN THE CRITICAL vortex configurations should be of HQV. REGION AT ZERO TEMPERATURE

2. 3D α-phases In this section, we study the collective modes in the Landau FL phase as the Pomeranchuk QCP is ap- proached, 0 > F a > 2 in 2D and 0 > F a > 3 in 3D. Similarly, the Goldstone manifold in the 3D α-phase l − l − for l = 1 can be written as These collective modes are the high partial wave channel counterparts of the paramagnon modes in the 3He

[SOL(3) SOS(3)]/[SOL(2) SOS(2) ⋉ Z2] system. The picture of collective modes in the p-wave ⊗ ⊗ channel at 2D is depicted in Fig. 6. = [S2 S2 ]/Z , (5.2) L ⊗ S 2 For this analysis, it is more convenient to employ the path integral formalism, and perform the where again the Z2 operation is a combined spin-orbit rotation at the angle of π to reverse the spin polariza- Hubbard-Stratonovich transformation to decouple the 4- tion and orbital distortion simultaneously. This Gold- fermion interaction term of the Hamiltonian presented in stone mode manifold gives rise to two Goldstone modes Eq.(2.4). After integrating out the fermionic fields, we in the density channel responsible for the oscillations of arrive at the effective action the distorted Fermi surfaces, and another two Goldstone 1 β modes for the spin precessions. The fundamental homo- S (~n )= dτ d~rd~r ′ (f a)−1(~r ~r′)~n (~r) ~n (~r′) 2 2 eff b l b b topy group of Eq. (5.2) reads π [S S ]/Z = Z . This −2 0 − · 1 L S 2 2 Z Z means that the π-disclination exists⊗ as a stable topologi- ∂ +tr ln + ǫ(~ ) ~n (~r) ~σg ( i~ ) . cal line defect. On the other hand, for the point defect in ∂τ ∇ − b · l,b − ∇ 3D space, the second homotopy group of Eq. (5.2) reads n o (6.1) π ([S2 S2 ]/Z ) = Z Z. This means both the orbit 2 L ⊗ S 2 ⊗ and spin channels can exhibit monopole (or hedgehog) In the normal FL state, we setn ¯ = 0, the fluctuations at structures characterized by a pair of winding numbers the quadratic level are given by the effective action (m,n). As a result of the Z2 symmetry, (m,n) denotes the same monopoles as that of ( m, n). 1 − − S(2) (n)= nµa(~q, iω ) L(FL) (~q, iω ) FL 2Vβ n µa,νb n ~q,iωXn B. Topology of the β-phases nνb( ~q, iω ) (6.2) × − − n In the 2D β-phase with l = 1 and w = 1 in the xy- where we have introduced the fluctuation kernel plane, the ground state is rotationally invariant, thus Lµa,νb(~q, iωn) which is given by 11

L(FL) (~q, iω )= (f a)−1(q)δ δ + Qµ,a(~q, iω )Qν,b( ~q, iω ) µa,νb n − 1 µν ab h n − − n iFL (6.3) and

µ,a ν,b 1 (FL) µ (FL) ν Q (~q, iω )Q ( ~q, iω ) = tr G (~k + ~q, iω ′ + iω )σ g (kˆ)G (~k,iω ′ )σ g (kˆ) . h n − − n iFL Vβ { n n a n b } ~ k,iωXn′ (6.4)

A. Two Dimensions

In 2D, without loss of generality we can choose the direction of ~q along the x-axis where the azimuthal angle θ = 0. The fluctuation kernels are given by ~q A: transverse modes (FL) 2 Lµa,νb(q,ω)= δabδµν κq + δ 1 n N(0) dθ s ab + A δab , 2 −1 2π s + iη cos θ Z − o 1 1 s = ω/(v q), δ = N(0) > 0, (6.7) F F a(0) − 2 | 1 | B: longitudinal modes where, as before, µ, ν = x,y,z are the components of the spin vector, and a,b = 1, 2 are the two orbital com- FIG. 6: The p-wave counterpart of paramagnon modes in 2D. ponents. The diagonal components of the angular form Taking into account the spin degrees of freedom, there are factors are Aaa = (cos2 lθ, sin2 lθ). For s< 1, the angular three (six) transverse triplet modes in 2D (3D), and three integral can be performed to yield longitudinal modes respectively. 2π dθ s cos2 lθ 2π s + iη cos θ sin2 lθ 0 µ,a Z s − is the correlation function of the Q operators, defined = 1 (s i 1 s2)2l . (6.8) in Eq.(2.3), in the FL phase (i.e. a fermion bubble). Here 2i√1 s2 ± − − − p For s 1, we can expand the above integral, and find, (FL) 1 ≪ G (~k,iωn)= (6.5) for l odd, iω ǫ(~k) n − (FL) 2 Lµa,νb(q,ω) = δabδµν κq + δ + N(0) is the fermion Green function in the FL phase, and it is { ls2 il2s3, for a =1 diagonal in spin space. − − . (6.9) × ls2 is, for a =2 After performing the Matsubara frequency summation, n − we find that the fluctuation kernel in the FL phase is Here, since we have chosen ~q along the x axis, the compo- given by the expression nent a = 1 denotes the longitudinal component (parallel to the direction of propagation ~q) and a = 2 is the trans- L(FL) (q,ω)= (f a)−1(q)δ δ verse component. Similarly, for l even, we get µa,νb − l µν ab d d k nf (ǫk−q/2) nf (ǫk+q/2) (FL) 2 − L (q,ω) = δabδµν κq + δ + N(0) + 2δµν d Aab, µa,νb (2π) ω + iη + ǫk−q/2 ǫk+q/2 { Z − ls2 is, for a =1 (6.6) − . (6.10) × ls2 il2s3, for a =2 n − − where we have performed an analytic continuation to real When l is odd, the transverse modes are overdamped frequency ω, and Aab is an angular form factor to be and the longitudinal modes are underdamped. For ex- defined below. ample, in the case of l = 1, the dispersion relation at the 12 critical point δ 0+ can be solved as where µ, ν = x,y,z are once again the three compo- → nents of the spin vector. For the l = 1 (p-wave) case κv q3 ω (q)= i F , (6.11) a,b =1, 2, 3 are the three orbital components. Using the 2 − N(0) formula for the transverse mode, and s +1 s +1 ln = ln iπΘ(s< 1), (6.14) s + iη 1 s 1 − κ 2 κ 3 − − ω1(q)= vF q i vF q . (6.12) we arrive at N(0) − 2N(0) r L(FL) (q,ω) = δ δ κq2 + δ for the longitudinal mode. In contrast, when l is even, µa,νb ab µν { such as the instability in the F s channel, the transverse 2 π 2 N(0)(s i 4 s), for a =1, 2 part is under-damped and the longitudinal part is over- + −2 π 3 , (6.15) N(0)(s + i 2 s ) for a =3 damped [4]. The difference is due to the different be- n − havior of the angular form factors for l even and l odd where only the leading order contribution to the real and respectively. imaginary parts are kept. The dispersion relation at the In both cases, just as it was found in Ref. [4], the critical point δ 0+ can be solved as dynamic critical exponent is z = 3. By the power count- → ing, the bare scaling dimension of the quartic terms in 3 the GL free energy, with effective coupling constants v 4vF κq 1 ω1,2(q)= i (6.16) and v , is (d + z) 4 where d is the spatial dimension, − π N(0) 2 − while the scaling dimension of the γ2 term linear in spatial derivatives is (d+z)/2 2. All of these operators are for the transverse modes, and irrelevant at zero temperature− in 2D and 3D. Thus, the critical theory is Gaussian, at least in perturbation the- κ π κ ω (q)= v q2 i v q3. (6.17) ory. However, it is possible that the above naive scaling 3 N(0) F − 4 N(0) F dimensional analysis may break down at the quantum r critical point. Various authors have found non-analytic corrections to Fermi liquid quantities at the ferromag- for the longitudinal mode. Similarly to the case in 2D, netic quantum critical point [56]. This may also occurs the longitudinal channel is weakly damped and other two here as well. We will defer a later research for the study transverse channels are over-damped. Again the dynamic of these effects. At finite temperatures, the critical region critical exponent z = 3, thus naively the critical theory turns out to be non-Gaussian [45]. Both the terms whose is Gaussian at the zero temperature. couplings are γ2 and v1,2 now become relevant. The rel- evance of the γ2 term does not appear in the usual ferromagnetic phase transitions [57], and we will also defer VII. THE GOLDSTONE MODES IN THE the discussion to this effect to a future publication. α-PHASE

B. Three Dimensions At the RPA level, the Gaussian fluctuations around the mean field saddle point of the α-phase are described In 3D, we can choose the z-axis along the direction of by an effective action of the form ~q. The diagonal part of the angular form factors is now 2 2 aa sin θ sin θ 2 1 A = ( , , cos θ). Assuming that q kF , the (2) µa (α) νb 2 2 Sα (n)= δn L (~q, iωn) δn . (7.1) fluctuation kernel can be approximated as | | ≪ 2Vβ µa,νb ~q,iωXn (FL) 2 Lµa,νb(q,ω)= δabδµν κq + δ The fluctuation kernel in the α-phase is 1 n N(0) s aa + d cos θ A , (α) a −1 µ,a 2 −1 s + iη cos θ Lµa,νb(~q, iωn) = (f1 ) (q)δµν δab + Q (~q, iωn) Z − o − h 1 1 ν,b s = ω/(v q), δ = N(0) > 0, Q ( ~q, iωn) α, (7.2) F F a(0) − 3 × − − i | 1 | (6.13) where 13

µ,a ν,b 1 (α) µ (α) ν Q (~q, iω )Q ( ~q, iω ) = tr G (¯n, k + q,iω ′ + iω )σ g (kˆ)G (¯n,k,iω ′ )σ g (kˆ) (7.3) h n − − n iα Vβ { n n l,a n l,b } ~ k,iωXn′

µa is the correlator of the operators Q in the mean ﬁeld y theory ground state of the α-phase (again a fermion bub- (α) ble). Here G (¯n,~k,iωn) is the fermion propagator in the α-phase with an expectation value of the (nematic- spin-nematic for the l = 2 case) order parameter equal ton ¯, ~q −1 G(α)(¯n,~k,iω )= iω ǫ (~k, n¯) (7.4) φ n n − α x where K~

ǫ (~k, n¯)= ǫ(~k) ~n ~σg (kˆ) α − h biα · l,b ~ ~ gl,1(k) = cos lθ~k, gl,2(k) = sin lθ~k, (7.5) is the fermion dispersion, a matrix in spin space, and ~nb α =n ¯ zˆ δb,1 is the mean field expectation value of FIG. 7: The azimuthal angle φ for the propagation wavevector hthei order parameter in the α-phase. ~q for the Goldstone modes in the 2D alpha phase. The spin Since in the α-phase there are spontaneously broken down and up Fermi surfaces can overlap each other by a shift at the wavevector K~ = 2¯n/vF . continuous symmetries, both in 2D and in 3D, the collective modes will consist of gapped longitudinal modes, i.e. along the direction of the condensate, and gapless, mode in the density channel, and two branches of Gold- Goldstone, modes transverse to the direction of sponta- stone modes in the spin channel.The fluctuation kernel neous symmetry breaking, both in the density and spin of the α-phases, L(α) is a 6 6 matrix. Its eigenval- channels. We will discuss the Goldstone modes in the µa,νb × β-phase in the next section. ues will thus yield six collective modes of which three are the above mentioned Goldstone modes. The other three We next comment on the stability of the α-phase in the p-wave channel. The GL energies Eq.(3.5) and Eq.(3.12) modes are gapped, and are associated with the structure of the order parameter in the α-phases. In the low fre- contain a cubic term linear in derivatives. In the ordered state, it might induce a linear derivative coupling be- quency regime ω n¯, we can neglect the mixing between the Goldstone modes≪ and other gapped modes. tween the massless Goldstone mode at the quadratic level through the condensate longitudinal mode, thus leading The density channel Goldstone mode is associated with the field n , conjugate to the bilinear fermion operator to a Lifshitz instability in the ground state. As we will z,2 show, this indeed occurs in the p-wave β-phase. However, Qz,2, is longitudinal in the spin sector and transverse in the charge sector, we will see that in the α-phase the spin (density) channel Goldstone modes have the same orbital (spin) indices as Q (~r)= ψ† (~r)[σ ( i )]ψ (~r), (7.6) those of the longitudinal mode, thus they can not be cou- z2 α z,αβ − ∇y β pled by Eq. (3.5) and Eq.(3.12). Instead, the Goldstone and describes the Fermi surface oscillation in the 2- modes couple to other gapped modes at the quadratic direction while keeping the spin configuration unchanged. level through linear derivative terms, which do not lead On the other hand, the spin channel Goldstone modes to instability at weak coupling, but can renormalize the nsp,x±iy, conjugate to the fermion bilinears Qx±iy,1 = stiffness of the Goldstone modes. (Qx,1 iQy,1)/2, describe spin oscillations while keeping the Fermi± surface unchanged.

A. The2D α-phases 1. Density channel Goldstone mode We consider the 2D α-phases assuming the order parameter conﬁguration as n =nδ ¯ δ . The order The density channel Goldstone ﬁeld n behaves sim- h µbi µz b1 z,2 parameter is thus the operator nz,1 of Eq. (2.1) and ilarly to its counterpart in the density channel Pomer- Eq. (2.2). As studied in Sec. IV A, the Goldstone mode anchuk instability [4]. The same approximation as in Ref. manifold S2 SO (2) results in one branch of Goldstone [4] can be used to deal with the anisotropic Fermi surface, × L 14

ω vF q i.e., keeping the anisotropy effect in the static part of the at n¯ , n¯ 1 in spite of the anisotropic Fermi surfaces, correlation function, but ignore it in the dynamic part. and the underdamping≪ of the Goldstone modes. The con- This approximation is valid at small values of order pa- tribution to the integral comes from the region around rameter, i.e., x =n/ ¯ (vF kF ) 1, wheren ¯ = nz,1 . We Fermi surfaces with the width about 2¯n cos(lθk)/vF . The define the propagation wavevector≪ ~q with theh azimuthali dependence of the integral on ~q can be neglected at vF q angle φ as depicted in Fig. 7 for the l = 1 case. n¯ 1 because a small ~q changes the integration area The effective fluctuation kernel for the charge channel weakly,≪ and it thus matters only for high order correc- Goldstone mode, which we label by fs, reduces to tions in ~q. The contribution to damping comes from the region where two bands become nearly degenerate, i.e., 2 2 is sin (lφ) ls cos(2lφ) cos(lθk) 0. However, the angular form factor also takes − the form≈ of cos(lθ ), which tends to suppress damping. (α) 2  (l even), k Lfs (~q, ω) = κq N(0) 2 2 As v q becomes comparable ton ¯, the anisotropy and − is cos (lφ)+ ls cos(2lφ) F  damping effects become more important. (l odd). The linear dispersion relation for the spin-channel  (7.7) Goldstone modes at vF q 1 holds regardless of whether  n¯ l is odd or even. This fact≪ is closely related to time re- where s = ω/vF q. Similarly to the results of in Ref. [4], versal (TR) and parity symmetry properties of the order this Goldstone mode corresponding to Fermi surface os- parameter nµa. For l odd, nµa is even under TR transfor- cillation is overdamped almost on the entire Fermi surface mation, and hence terms linear in time derivatives cannot except on a set of directions of measure zero: for l even, not appear in the effective action. On the other hand, for the charge channel Goldstone mode is underdamped in l even even, although nµa is odd under TR, in 2D we can the directions φ = nπ/l, n =0, 1,...,l, which are just the still define the combined transformation T ′ as symmetry axes of the Fermi surface; for l odd, it is underdamped instead in the directions φ = π(n +1/2)/l, and T ′ = T R(π/l), (7.12) in this case the Goldstone mode is maximally damped along the symmetry axes. under which nµa is even. Here R(π/l) is a real space rotation by an angle of π/l. thus, also in this case, terms which are linear in time derivative are not allowed in 2. Spin channel Goldstone modes the effective action. In contrast, for the case of a ferromagnet at l = 0, TR symmetry is broken, and no other On the other hand, the spin channel Goldstone fields symmetry exists to form a combined operation T ′ that nx±iy,1 behave very differently. They only involve “inter- will leave the system invariant. As a result, terms lin- band transitions”, leading instead to a fluctuation kernel ear in time derivatives appears in the effective action of of the form a ferromagnet. The same arguments apply for phases with mixed ferromagnetic and spin nematic order (and its 2 2 1 d k 2 generalizations). Furthermore, in the presence of time- Lx+iy,1(~q, ω)= κq + a +2 2 cos (lθk) f1 (2π) reversal-violating terms, the two transverse components | | Z of spin fluctuation become conjugate to each other as in n [ξ (~k ~q/2)] n [ξ (~k + ~q/2)] f ↓ − − f ↑ , (7.8) the presence of ferromagnetic long range order. In this ~ ~ × ω + iη + ξ↓(k ~q/2) ξ↑(k + ~q/2) case, only one branch of spin wave Goldstone mode exists − − 2 with a quadratic dispersion relation ωF M q . where ξ1,2(k) = ǫ(~k) µ n¯ cos(lθk). They satisfy the ∝ relation of − ∓ 3. Spin wave spectra L (~q, ω)= L ( ~q, ω). (7.9) x−iy,1 x+iy,1 − − a A detailed calculation, presented in Appendix B, shows We assume that the F0 channel is off-critical, thus in that for ω , vF q 1, the kernel L reads the normal state no well-defined spin wave modes ex- n¯ n¯ ≪ x±iy,1 ist. However, in the α-phase the spin channel Goldstone 2 modes carry spin, thus induce a well-defined pole in the N0 ω L (~q, ω)= κq2 , (7.10) spin wave spectrum. This can be understood from the x±iy,1 − 2 F a n¯2 | l | commutation relation between spin modes and the spin which gives rise to a linear and undamped spectrum: channel Goldstone modes

a [Sx iSy,Qx∓iy,1] = 2Qz,1. (7.13) 2κ Fl ± ± ωx±iy,1(~q)= | | n¯ ~q . (7.11) s N(0) | | In the α-phase where Qz1 obtains a non-vanish expectation value, then these two channels become conjugate. Eq. (7.11) has to two important features (due to the As a result, the spin-wave gains a sharp resonance and interband transition): the isotropy of dispersion relation should exhibit in the neutron scattering experiment. In 15 contrast, in the normal state, the coupling between this It is overdamped almost everywhere, except if ~q lies in two modes is negligible, and thus the resonance disap- the equator (θ = π/2), in which case it is underdamped pears. A similar physics occurs in the SO(5) theory for and has a quadratic dispersion. On the other hand, the the explanation of π-resonance in the underdoped high fluctuation kernel of the Goldstone mode of nz,2 reads Tc cuprates [58]. π The effective coupling constant which mixes the S + L = κq2 iN(0)s , (7.19) x z,2 − 4 iSy and Qx−iy,1 operators is a bubble diagram which can be calculated as which has no dependence on the angle θq. Hence, this ω mode is over-damped on the entire Fermi surface. 0 a χs(~q, ω)= N(0) . (7.14) The spin channel Goldstone modes of n in the F − n¯ x±iy,3 1 channel behaves similarly to that in the 2D case. We sim- a This bubble is dressed by the interaction in the Fl chan- ply present their fluctuation kernels at small wavevectors nel. The resonant part of the spin correlation function as (i.e. the contribution of the collective mode pole) be- 3N(0) comes L (~q, ω) = κq2 s2 (7.20) x±iy,3 − F a 0 2 | 1 | χs(~q, ω) χs(~q, ω) = S+(~q, ω)S−( ~q, ω) = | | where s = ω/vF K . The spin wave excitation is also h − − i Lx+iy,1(~q, ω) | | a 2 dressed by the interaction in the F1 channel in the α- ω N(0) n¯2 phase. By a similar calculation to the 2D case, we have = 2 . (7.15) κq 2 ω2 a 2 iδ N(0) |Fl | n¯ χ (~q, ω) = S (~q, ω)S ( ~q, ω) − − s h + − − − i χ0(~q, ω) 2 For fixed but small ~q, the spectral function exhibits a = | s | . (7.21) δ-function peak at the dispersion of the collective mode Lx+iy,3(~q, ω) 2 2 2 a 2 2 2 Thus it also develops the same pole as in the spin channel Imχs(~q, ω)= κπvF q n¯ F1 δ(ω ωq ). (7.16) | | − Goldstone modes. which will induce a spin resonance in all directions. It is worth to note that in the spin channel the isotropy in this dispersion relation at small ~q persists even deeper in VIII. GOLDSTONE MODES IN THE β-PHASES the ordered phase. In this section, we calculate the Goldstone modes and spin wave spectra in the β-phase at 2D and 3D at the B. The3D α-phase for l = 1 RPA level. We will show that for l = 1, a Lifshitz-like instability arises leading to a spatially inhomogeneous In the 3D α-phase, we assume the Fermi surface distor- ground state. This is because of a dynamically generated tion along the z-axis, and the order parameter configura- Dzyaloshinskii-Moriya interaction among the Goldstone µa tion as n =nδ ¯ µ,zδa,3. Similarly to the 2D case, the spin modes as a result of the spontaneously breaking of parity. up and down Fermi surfaces are related by a overall shift A G-L analysis is presented to analyze this behavior. at the wavevector K~ = 2¯n/vF zˆ. The remaining symmetry is SOL(2) SOS (2) which results in four Goldstone modes. They⊗ can be classified as two density channel A. The2D β-phases modes and two spin channel modes. Without loss of generality, we choose the propagation wavevector ~q lies in Without loss of generality, we consider the β-phase a the xz-plane. in the Fl channel. We first assume a uniform ground The density channel Goldstone modes describe the state with the configuration of the d-vector d~ = Fermi surface oscillations in the x and y directions, which (cos lθ, sin lθ, 0) as defined in Eq. (4.6). The correspond- are associated with the fields nz,1 and nz,2. By a Leg- ing order parameter i.e., the Higgs mode nhiggs, is con- endre transformation, they are conjugate to the bilinear jugate to operator Ohiggs as operators Qz,1 and Qz,2: 1 † Ohiggs(~r)= Qx,1(~r)+ Qy,2(~r) . (8.1) Qz1(~r) = ψ (~r)[σz,αβ( i x)]ψβ(~r) √2 α − ∇ Q (~r) = ψ† (~r)[σ ( i )]ψ (~r) (7.17) z2 α z,αβ − ∇y β By performing a relative spin-orbit rotation around the z, x, and y-axes on the mean field ansatz, we obtain the Following the same procedure of the calculation in 2D, operators for three branches of Goldstone modes as we find that the fluctuation kernel of the Goldstone mode nz,1 is 1 Oz(~r) = (Qx,2(~r) Qy,1(~r)), √2 − 2 π 2 2 Lz1 = κq i N(0)s cos θq + N(0)s cos2θq. (7.18) O (~r) = Q (~r), O (~r)= Q (~r). (8.2) − 4 x − z,2 y z,1 16

~q y′ x′ reﬂection symmetry even in the presence of the ~q, θ 2φ θ , σ σ , k → − k z →− z σ σ cos2lφ + σ sin2lφ, z x → x y σy σx sin2lφ σy cos2lφ. (8.4) φ ~s → − Oz and Oy′ are even under this transformation while Ox′ is odd. Thus, Ox′ decouples from Oy′ and Oz, while hybridization occurs between Oz and Oy′ . For a small vF q ω wavevector n¯ 1 and low frequency n¯ 1, we can ignore the mixing≪ between the Goldstone modes≪ and other gapped modes. For l 2, the eigenvalues of the ﬂuctua- tion kernel for the Goldstone≥ modes is

l = 1, w = 1 Lzz(q,ω) Lx′x′ (q,ω) Ly′y′ (q,ω) ≈ ≈ ω2 N(0) κq2 , (8.5) ≈ − 4¯n2 F a FIG. 8: The three Goldstone modes of Oz , Ox′ , Oy′ of the 2D | l | β-phase can be viewed as a relative spin-orbit rotation for this where we have neglected the anisotropy among the three phase with angular momentum l = 1 and winding number dispersion relations. A finite hybridization between Oz w = 1. Here φ is the azimuthal angle of the propagation l direction ~q. and Oy′ appears at the order of O(q ) l Lzy′ (q,ω) iq . (8.6) ω ∝ 2¯n + vF q which is negligible at small q at l > 2. Thus, the spectrum of the Goldstone modes is linear for l > 2. For l = 2, the hybridization is quadratic in ~q particle-hole 2¯n N(0) 2 2 L ′ (q,ω)= i q (1+4b) κq , (8.7) continuum zy 2 − 32√2kF ≪ and thus must be taken into account. The resulting 2¯n − vF q eigenmodes in the transverse channel are O iO ′ . How- z ± y ω = vgsq ever, the linear dispersion relation remains at l = 2. Similarly to ferromagnets, in there are two Fermi sur- q′ q faces with unequal volume in the β-phases. The interband transition has a gap of 2¯n and a particle-hole continuum of width 2vF q as depicted in Fig. 9. The Gold- FIG. 9: The linear dispersion relation of Goldstone modes of stone modes correspond to the interband transition with the 2D β-phases (l ≥ 2) at small momentum ~q. For q > q′, a a velocity vgs 2¯n κ Fl (0) /N(0), and no Landau- the Goldstone modes enter the particle-hole continuum, and damping effects≃ exist at| small |q. Naturally, after Gold- are damped. stone modes enters thep particle-hole continuum, at the ′ wavevector q 2¯n/(vgs + vF ), the mode is no longer long lived and≈ become Landau damped. We define the propagation wavevector of Goldstone The linear dispersion relation of the Goldstone modes modes ~q and its azimuthal angle φ as depicted in Fig. holds for all the values at l 2. This feature is also due 8. For the general direction of ~q, it is more convenient to to the symmetry properties≥ of the nµa under TR and set up a frame with three axes x′, y′ and z, where x′ ~q parity transformation. The reasoning here is the same as ′ k and y ~q. We rotate Ox and Oy into that for the α-phase in the Section VII A. ⊥ We next calculate the spin-wave spectra in the β-phase

Ox′ = cos lφ Ox sin lφ Oy at l 2. We have the following commutation relations − as ≥ Oy′ = sin lφ Ox + cos lφ Oy. (8.3) iφq −iφq [S+e ,e (Ox iOy)/√2] = iOhig + Oz, ′ ′ − Oz, Ox , and Oy are the generators of a relative spin- [Sz, Oz]= iOhig. (8.8) orbit rotation around the z, x′, and y′-axis respectively. Thus, in the following, we call the Goldstone mode of The eﬀective coupling between Sz and Oz can be calcu- Ox′ the longitudinal Goldstone mode, and those of Oz lated as and Oy′ are two transverse Goldstone modes. i ω N(0) χ0 (~q, ω)= − + O(q2). (8.9) The system with d~ = (cos lφ, sin lφ, 0) has the following Sz √2 n¯ F a | l | 17

2 eˆy eˆy ω ′ ′ Oz−iy′ Ox Oz+iy ~n2 ~n2

eˆx ~n1 ~n1 eˆx

qc q eˆz ~n3

a 2¯n|f1 |hso A B

FIG. 11: Longitudinal and transverse twists for the Lifshitz- like instability in the β-phase with l = 1. A) The longitudinal FIG. 10: [color online] The dispersion relation of Goldstone twist withn ˆ2 processing around thee ˆ1 axis as described in modes for the β-phase with l = 1. The longitudinal modes Eq. (8.20). B) The transverse twist with the triad formed by Oy′ has linear dispersion relation, while the two transverse nˆ1,n ˆ,2, andn ˆ3 =n ˆ1 × nˆ2 precessing around thee ˆx axis as modes Oz±iy′ are unstable towards the Lifshitz-like instability described in Eq. (8.24). at small momentum ~q.

scales linearly with q: Thus the spin-spin correlation function, dressed by the a Fl channel interactions, near the resonance of the dis- i 3 q persing transverse collective mode has the form L ′ (~q, ω)= L ′ (~q, ω) b + N(0)x . zy − y z ≈ √ 2 k 4 2 F 0 2 χs (8.15) χs(~q, ω) = Sz(~q, ω)Sz( ~q, ω) = | | h − − i Lzz(~q, ω) 2N(0) ω2 We diagonalize the matrix and then obtain the eigen- ′ = a |F a| . modes as Oz iOy with the following dispersion relation Fl ω2 4¯n2 l κq2 ± | | − N(0) 2 (8.10) 2 2 a κq q ωT,± = 4¯n F1 (0) cx , (8.16) | | N(0) ± kF The spectral function at the resonance reads as depicted in Fig. 10 where c is a constant at the order Imχ (~q, ω)=8¯n2κq2δ(ω2 ω2). (8.11) s − q of 1. Clearly, ω2 < 0 for small ~q . This means that the Similarly, the effective coupling between S eiφq and T,− + uniform ground state can not be| stable| in the F a channel e−iφq (O iO )/√2 gives the same result, 1 x − y due to a Lifshitz-like instability [59, 60]. This instabil- 0 0 ity can be understood in terms of the nontrivial effects χs ± = χsz (8.12) x iy of the gradient term with coefficient γ2 in the G-L free energy in Eq.(3.5). this term is cubic in the order pa- The transverse spin-spin correlation function is rameter nµ,b and a linear in spatial derivatives. This 0 2 term leads to an inhomogeneous ground state in the β- χsx±iy S+(~q, ω)S−( ~q, ω) = | | (8.13) phase in which parity is spontaneously broken. A sim- h − − i Lx+iy,x−iy ilar phenomenon occurs in the bent-core liquid crystal system where a spontaneously chiral inhomogeneous ne- which is also dressed by the interactions. matic state arises in an non-chiral system. A similar term involving a linear derivative and the cubic order of order parameter is constructed in Ref. [61] to account for this B. Lifshitz-like instability in the 2D p-wave channel transition. These inhomogeneous ground states also occur in chiral liquid crystal [6] and helimagnets [43] such For l = 1, with the assumption of the uniform ground as MnSi in which parity is explicitly broken. In contrast, state, the dispersion for the longitudinal mode Oz re- such a term is prohibited in the ferromagnetic transition mains linear as [44, 45], and the nematic-isotropic phase transition [4] in κ the density channel Pomeranchuk instabilities. We will ω2 = 4¯n2 F a q2. (8.14) L | 1 |N(0) see that the G-L analysis below based on the symmetry argument agrees with the RPA calculations in Eq.(8.6). However, the situation for the transverse modes is dra- In order to obtain the values of γ1,2, we linearize the matically different. The mixing between Oz and Oy′ gradient terms in Eq. (3.5) in the ground state with d- 18 vector configuration of d~ = (cos θ, sin θ, 0), i.e., transverse mode Oz + iOy describes the precession of the triad ofn ˆ ,n ˆ ,n ˆ =n ˆ nˆ as 1 2 3 1 × 2 2 2 ~n1 = n¯ δn1eˆx + δ~n1, − ~n1 =n ¯ cos ǫ eˆx + sin ǫ cos qx eˆy sin ǫ sin qx eˆz , q − 2 2 2 ǫ 2 ǫ ~n2 = n¯ δn2eˆy + δ~n2, (8.17) ~n =n ¯ sin ǫ cos qx eˆ + (cos sin cos2 qx)e ˆ − 2 − x 2 − 2 y q 2 ǫ where δ~n1 eˆx, δ~n2 eˆy and ~n1 ~n2 is kept. The sin sin2qx eˆz , (8.24) contribution⊥ from the Goldstone⊥ modes⊥ can be organized − 2 into where ǫ describes the precession amplitude; ~n1 precesses around the x-axis, and ~n2 traces a “figure of eight” orbit ~ 2 2 2 Fgrad(O) = γ1 (∂iOx) + (∂iOy) + (∂iOz) on the surface of the unit sphere. To linear order of ǫ, { } γ2n¯ the free energy cost is + (Ox∂yOz Oy∂xOz) √2 − 3 n¯ h 3 2 V (n)= γ q2(¯nǫ)2 + qγ (¯nǫ)2 + so (¯nǫ)2. (8.25) + γ2n¯ (∂xOx + ∂yOy). (8.18) 2 1 2 2 n¯ 4

With the assumption of the uniform ground state, the Hence, when hso is larger than a critical value hso,T = 2 3 first γ2 term behaves like a Dzyaloshinskii-Moriya term, γ2 n¯ /18γ1 < hso,L, the instability due to the transverse and gives a liner dependence in the dispersion relation as Goldstone modes is suppressed. appears in Eq.(8.16). By matching the coefficients, we Because hso,L > hso,T , the longitudinal Goldstone arrive at the result channel instability is stronger than that in the transverse channel. Thus, in the absence of the external field, the √2cN(0) ground state exhibits the longitudinal twist of Eq.(8.20). γ1 = κ, γ2 = 2 . (8.19) vF kF However, this spiral order can not give static Bragg peaks in neutron scattering experiments as occurs in the case of The second coefficient γ2 in the GL expansion of a helimagnet, although it can couple to the spin-spin cor- Eq.(8.18) is a total derivative of the longitudinal Gold- relations through a dynamic effect. However, the spatial stone modes. It does not contribute to the equation of inhomogeneity complicates the calculation. This remains motion for the Goldstone modes around the saddle point an interesting problem which we will not pursue further of the uniform mean field ansatz. However, Eq.(3.5) ac- in this work. tually allows a twisted ground ground state in the longi- If hso hso,L, then both instabilities are suppressed, tudinal channel as depicted in Fig. 11 A. Without loss and thus≥ in this regime the ground state is uniform. In of generality, we assume a pitch vector along the x axis, this case, the Goldstone spectrum has an overall shift ~q xˆ, and perform a longitudinal twist of the Goldstone k based on Eq.(8.18) as configuration of ~n1,2 as ′2 2 a ′2 2 a ωl = ωl +2 f1 nh¯ so, ωt = ωt +2 f1 nh¯ so. (8.26) ~n1 =n ¯(1, 0, 0), ~n2 =n ¯(0, cos qx, sin qx). (8.20) | | | | − as depicted in Fig. 10. Then the spectra for the two This configuration means that we fix ~n1, and rotate ~n2 transverse Goldstone modes exhibit a roton like structure around the x-axis. If the system has a small external with a gap of spin-orbit coupling, it pins the order parameter configuration. We introduce an effective spin-orbit field h to a so ∆= 2¯n f1 (hso hso,T ), (8.27) describe this effect | | − q at q′ = γ n/¯ (4√2γ ). V (h )= h (n + n ). (8.21) c 2 1 so − so x,1 y,2 Then the G-L free energy becomes C. Goldstone modes in the 3D β-phase 2 2 3 F (n)= γ1n¯ q γ2n¯ q hson¯ cos qx. (8.22) − − For the 3D β-phase, we only consider the case of l =1 If hso is less than a critical value hso,L defined as with the ground state configuration of the d-vector as d~(~k) ~k. The Goldstone modes behave similarly to the 2 3 k γ2 n¯ 2D case. The Legendre conjugation operators of the or- hso,L = , (8.23) µ,b 4γ1 der parameter n can be decomposed into the operators O (J =0, 1, 2; J = J, ...J) as eigen-operators of the J,Jz z − then the Lifshitz instability occurs with the pitch of qc = total angular momentum J~ = L~ + S~ in the β-phase. The γ2n/¯ (2γ1). If hso > hso,L, the instability due to the Higgs mode carries J = 0 defined as longitudinal Goldstone mode is suppressed. Now let us look at the instability caused by the trans- 1 µa Ohig(~r)= δµan (~r). (8.28) verse Goldstone modes as depicted in Fig. 11 B. The √3 19

Due to the broken relative spin-orbit symmetry, the relative spin-orbit rotations generate the three Goldstone modes Ox, Oy, and Oz B~ 1 µ,a Oî(~r) = ǫiµan (~r), (8.29) √2 which carry total angular momentum J = 1. We choose the propagation wavevector ~q along the z-axis. Due to l = 1 l = 2 presence of ~q, only Jz is conserved. The relation between Ox, Oy and Oz and those in the helical basis O1,±1 and O10 is FIG. 12: The α-phases in a B~ field (l = 1, 2). The larger Fermi surface has spin parallel to the B~ field. The Fermi 1 surface distortion is not pinned by the B~ field. O1,±1 = (Ox iOy), O1,0 = Oz. (8.30) √2 ± In the low frequency and small wavevector regime as where the difference among the three Frank constants is ω, vF q n¯ vF kF , we can neglect the mixing between neglected. Goldstone≪ modes≪ and other massive modes. Similarly to Following the same procedure in the 2D case in Section the case in 2D, we calculate the fluctuation kernels of the VIII B, we study the instabilities of the longitudinal and Goldstone modes as transverse twists in 3D β-phase with an external spin-

2 orbit field to pin the order parameter 2 ω L10,10(q,ω) = κq , (8.31) 3D x,1 y,2 z,3 2 a Vso = hso (n + n + n ). (8.37) − 4¯n f1 − | | 2 2 N(0) q ω After straightforward calculation, we find that longi- L1 1 1 1(q,ω) = κq x . (8.32) ± , ± 18 k 4¯n2 f a tudinal twist occurs at the pitch wavevector qL,3D = ± F − 1 3D | | nγ¯ 2/(2γ1) with a critical value of hso field to suppress the 3D 2 3 For simplicity, we have neglected the anisotropy in the twist at hso,L = γ2 n¯ /(4γ1), and those of the transverse 2 3D q term in Eq. (8.32). Their spectra read twist are qT,3D = qL,3D/2 and hso,T = hso,L/2. Thus the conclusion is the same as in the 2D case that the instabil- κq2 q x ity of the longitudinal twist is stronger than that of the ω2 = 4¯n2 F a + j | | (j =0, 1). (8.33) jz | 1 | N(0) z 18k z ± transverse twist. Again, if h3D h3D , then the ground F so ≥ so,L state is uniform, and the spectrum of the two transverse Because of the broken parity, the channel of jz = 1 Goldstone modes exhibits a roton-like structure with a − is unstable, which leads to the Lifshiz-like instability as gap of discussed in the 2D β-phase. Again this Lifshitz instability is due to the nontrivial ∆= 2¯n f a (h h ), (8.38) | 1 | so − so,T γ2 term in the G-L free energy of Eq.(3.12). To determine located at q . q the coefficients of the gradient terms, we linearize the γ2 T,3D term around the saddle point, define the deviation from the uniform mean field ansatz as IX. MAGNETIC FIELD EFFECTS AT ZERO TEMPERATURE ~n = n¯2 δn2eˆ + δ~n , ~n = n¯2 δn2eˆ + δ~n , 1 − 1 x 1 2 − 2 y 2 q q 2 2 In this section, we discuss the magnetic field effect to ~n3 = n¯ δn eˆz + δ~n3, (8.34) − 3 the order parameter configurations in the α and β-phases. q Due to the symmetry constraint, the B~ field does not cou- where δ~n1 eˆx, δ~n2 eˆy, and δ~n3 eˆz. The contribu- µa tion from the⊥ Goldstone⊥ modes becomes⊥ ple to the order parameter n linearly, and the leading order coupling begins at the quadratic level as ′ 2 2 2 Fgrad(O) = γ1 (∂iOx) + (∂iOy) + (∂iOz) ~ 2 2 T 2 µ ν µb νb { } ∆F (B, ~nb) = (u + w)g B tr[n n] wg B B n n ′ − γ2 ′ 2 2 2 2 2 2 nǫ¯ ijkOi∂j Ok + γ √2¯n ∂iOi. (8.35) = ug B n + wg (B~ ~n ) , (9.1) − 2 2 b × b b b ′ ′ X X Similarly to the 2D case, the values of γ1 and γ2 can be with g the gyromagnetic ratio. The coefficients u and w determined by matching the coefficients of the dispersion can be determined from the microscopic calculation by relation in Eq.(2.5) as using the mean field approach as

N(0) u = 4v1 2v2, w =4v2 (at 2D), ′ ′ − γ1 κ, γ2 = 2 , (8.36) ≈ 18v k u = 6v1 3v2, w =5v2 (at 3D and l = 1), (9.2) F F − 20

B~ rotations, i.e. a spacial rotation followed by a spin reversal, while retaining the symmetry under rotations by π. ~n2 Thus, the magnetic field induces a non-zero component ~n1 of the charge nematic order parameter, the nematic in the spin-singlet channel. (Analogs of this effect hold in all angular momentum channels). On the other hand, for a translation and rotationally invariant system, the Fermi surface distortion, i.e. orientation of the order parameter in real space, cannot be locked to the direction of the B~ field. As a result, the Goldstone manifold becomes [SO (2) SO (2)]/SO (2) = SO (2). FIG. 13: The spin configuration on the Fermi surfaces in the S ⊗ L S L a ~ The B field in the β-phase also constrains the direction β-phase (Fl channel). With the B field, the large circle splits ~ of the vectors ~nb. Since w is negative in the β-phase, to two small circles perpendicular to the B field with the de- ~ velopment of a finite magnetization. The large (small) Fermi B prefers to be perpendicular to ~nb vectors. Thus in ~ surface polarizes parallel (anti-parallel) to the B~ field. the 2D system, ~n1, ~n2, and B form a triad which can be either left-handed or right-handed as depicted in Fig. 13. The spin configuration on Fermi surfaces changes ~ ~n3 B from a large circle into two smaller circles with a net ~n2 ~ ~n1 B~ spin polarization along the B field. The triad still has the ~n3 degrees of freedom to rotate around the axis of B~ , thus ~n2 the Goldstone manifold at 2D is reduced from SOL+S(3) to SOL+S(2) Z2. The B field effect in 3D systems is ~n1 × more subtle. At small values of B, ~n1, ~n2 and ~n3 can no longer for a triad with B~ . Instead, ~n1, ~n2 and ~n3 for a distorted triad with the direction of the distortion lying (a) (b) in the diagonal direction and parallel to B~ , as depicted in Fig. 14. A. As B grows larger, the three vectors ~n1, ~n2 and ~n3 are pushed towards the plane perpendicular to FIG. 14: The order parameters in the 3D β-phase at l = B~ . For B larger than a value B′, defined as 1. a) Configuration for B′ > B as defined in Eq. (9.5); b) ′ v Configuration for Bc >B>B . B′ = B 2 , (9.5) c v +4 w v /u r 2 | | 1 ~n , ~n and ~n become co-planar, and with a relative angle with v defined in Eq.(4.12) in 2D and Eq.(4.19) in 3D 1 2 3 1,2 of 2π/3, i.e. an equilateral triangle, as depicted in Fig. respectively. v1 is usually larger than v2, thus u is typi- ~ 14 B. If B is further increased, ~n1, ~n2 and ~n3 keep these cally positive. This means that the external B field sup- directions but their magnitudes continuously shrink to presses the magnitude of order parameters as zero at B = Bc. We next discuss the effect of magnetic field B on the B2 Lifshitz-like instability in the 2D β-phase at l = 1. We n¯(B)=¯n(B = 0) 1 2 (9.3) s − Bc assume that B~ is parallel to the z-direction. As showed in Section VIII B, the instability of the longitudinal twist in both α and β-phases, where the critical value of Bc is described in Eq. (8.20) is stronger than that of the trans- defined as verse twist. Thus, we will focus here on the case of the longitudinal twist. In this case, the order parameter ~n2 |r| precesses around thee ˆ axis. Thus, B~ , ~n and ~n cannot µBBc u n¯(0) 1 1 2 = = 1. (9.4) form a fixed triad uniformly in space. The free energy of v k vqk k v ≪ F F F F F F Eq. (3.5) now becomes: This effect has been studied in Ref. [32]. 2 2 3 2 2 2 V (n) = γ1n¯ q γ2n¯ q wB n¯ (1 + cos qx). (9.6) In the α-phase, w is positive. Thus, the spin compo- − − nents of the order parameter of this phase, ~nb, prefers Hence, for B larger than a critical value of Bcl, ~ to be parallel or anti-parallel to the magnetic field B. Bc Consequently, the Fermi surface with its spins polarized Bl = , (9.7) 1+8v w γ /(uγ2) parallel to the magnetic field becomes larger in size than 1| | 1 2 the Fermi surface for the spins pointing in the opposite the Lifshitz instabilityp of the β-phase to a phase with a direction. In the case of the nematic-spin-nematic phase, longitudinal twist is suppressed. The effects of an exter- the α-phase with angular momentum l = 2, the main ef- nal B field in the 3D β-phase are more complicated and fect of the magnetic field is to break the symmetry by π/2 will not be discussed here. 21

Finally, we note that the order parameter nµa can cou- along the major and minor axes induces a spin current µ,a ple to the magnetic field B linearly when other external Js flowing in the same (or opposite) direction. But a fields are also present. For l = 1, a possible additional for the general direction of Jc , the induced spin current µ,a a term in the free energy of the form Js flows with an angle with Jc . We denote the az- a imuthal angle between the charge current Jc and the ∆F (B, j)= γ Bµj (~r)nµ,a, (9.8) a µ,a 3 a x-axis as φ. Then the angle between Jc and Jc reads 2φ or π 2φ depending on the sign of g. The nature where ja is the a-th component of the electric current. of the induced− spin current here is different from that To leading order, the mean-field value of the coupling of the spin-Hall effect in semi-conductors with SO cou- constant γ3 is pling. In that case, the spin-Hall current always flows 2 perpendicular to the electric field, and the spin Hall con- ~ eµB N(0) γ3 = ∗ , (9.9) ductance is invariant under time-reversal transformation. −2m vF vF kF Here, because of the anisotropy of the Fermi surfaces, where m∗ is the effective mass and e is the charge of the spin current is perpendicular to the charge current, electrons. For l = 2, a similar term can be constructed only if the charge current flows along the diagonal direc- as tion (φ = π/4, 3π/4). On the other hand, the d-wave phases break± time± reversal symmetry, thus the induced µ µ,a ∆F (B,u)= γ4B ua(~r)n , (9.10) spin current is not dissipationless. where ua is the strain field. Due to these terms, in the α- In the β-phase, without loss of generality, we can take phase, the electric current and lattice strain can be used order parameter configuration as in Fig. 3. a, i.e., to lock the direction of the order parameter in real space in the presence of an external magnetic field at l = 1 and l = 2 respectively. In the β-phase, this term will distort eˆ eˆ nµ,ab =n ¯ x y , (10.4) the round Fermi surfaces into two orthogonal ellipses but eˆy e¯x with different volume. −

wheree ˆx,y denote the spin direction. We assume that X. SPIN CURRENT INDUCED BY A CHARGE a the charge current Jc flows along the x direction. Then CURRENT IN THE d-WAVE CHANNEL two spin currents polarizing along orthogonal directions, are induced with the same magnitude. The spin current In the d-wave (l = 2) case, the order parameters have flowing along the x-direction polarizes alonge ˆx, while the structure of the spin-quadruple moments. From the that flowing along the y-direction polarizes alonge ˆy. If µ,a symmetry analysis, a spin current Js may be induced we measure the spin current along the spatial direction a by a charge current Jc flowing through the system where with the azimuthal angle φ respect to the x-axis, the µ is the spin index, and a,b are the spatial indices. For induced spin current along this direction polarizes along simplicity, we only study the 2D α and β-phases. In the the direction of cos φ eˆx + sin φ eˆy. Because in the β- standard quadruple notation, the order parameters nµ,1 phase, there is an induced SO coupling, spin and orbital and nµ,2 can be represented as nµ,1 = nµ,xx nµ,yy, and angular momenta will not be preserved separately. As a µ,2 µ,xy µ,yx − n =2n =2n . Then we write the formula as result, it is impossible in the β-phase to describe a spin

µ,a µ,ab b current by two separated indices (a spatial index and a Js = gn¯ Jc , (10.1) spin one). The spatial degrees of freedom and the spin ones must be mixed together. where the matrixn ¯µ,ab is related to the order parameter n¯µ,a as Finally, in a real material, there would always be some SO coupling. With even an infinitesimal SO coupling, a n¯µ,xx n¯µ,xy n¯µ,1 n¯µ,2 n¯µ,ab =2 = . (10.2) charge current flowing inside the system can remove the n¯µ,yx n¯µ,yy n¯µ,2 n¯µ,1 − degeneracy of the ground states in the ordered phase. In By the standard the linear response theory, the coefficient other words, in the presence of explicit SO interactions, g can be calculated as a charge current can pin down the direction of the order parameter. Therefore, the relative angle between the (3 2a)π order parameter and the charge current is not arbitrary. g = − a . (10.3) As a result, to be able to adjust the angle between the ekF vF F2 | | charge current and the order parameter as we mentioned In the α-phase, it is convenient to choose the direc- above, some other mechanism is necessary to pin down tion of the axes of the reference frame x and y along the order parameter such that the order parameter will the major and minor axes of the distorted Fermi sur- not rotate when we rotate the direction of the charge cur- faces, and assume spin quantization along z-axis, so that rent. For example, an in-plane magnetic field or a lattice n¯µ,ab =n ¯eˆ diag 1, 1 . A charge current J a running potential as background can do the job. z { − } c 22

XI. POSSIBLE EXPERIMENTAL EVIDENCE the specific heat jump, and more importantly the jump FOR THESE PHASES of the non-linear spin susceptibility χ3 at the transition. The origin of the random field in the NMR experiment is At present time, we are not aware of any conclusive explained by the spin moment induced by disorder in the experimental evidence for a spin triplet channel Pomer- p-wave α-phase. However, the α-phase still has Fermi anchuk instability. However, the α and β-phases pre- surfaces, thus it is difficult to explain the large loss of sented here are just a natural generalization of ferromag- density of states. Further, the α-phase is time reversal netism to higher partial wave channels. Taking into ac- even, thus its coupling to spin moment must involve B count the existence of the p-wave Cooper pairing phase field. In the NMR experiment, an external B field is in- in the 3He systems[1] and the strong evidence for its ex- deed added. It would be interesting to check whether the line-shape broadening is correlated with the magnitude istence in the ruthenate compound Sr2RuO4 [62], we be- lieve that there is a strong possibility to find these phases of B. in the near future. Basically, the driving force for behind c. Sr3Ru2O7 : The bilayer ruthenate compound the Pomeranchuk instabilities in the spin triplet channels Sr3Ru2O7 develops a metamagnetic transition in an ap- is still the exchange interaction among electrons, which plied magnetic field B perpendicular to the c-axis. In shares the same origin as in ferromagnetism. Although very pure samples, for B from 7.8 T to 8.1 T , the resis- the weak coupling analysis we have used here may not tivity measurements show a strong enhancement below apply to the materials of interest, as many of them are 1.1 K [14]. Transport measurements in tilted magnetic strongly correlated systems, many of the symmetry is- fields, with a finite component of the B field in the ab sues will be the same as the ones we have discussed here, plane, shows evidence for a strong in-plane temperature- with the exception of the role of lattice effects which we dependent anisotropy of the resistivity tensor, which is have not addressed in detail and which may play a sig- suppressed at larger in-plane fields[14, 15]. This effect is nificant role, i.e. by gapping-out many of the Goldstone interpreted as a nematic transition for the Fermi surface modes associated with the continuous rotational symme- of the majority spin component [14]. This result suggests try of the models that we have discussed. Nevertheless a state which is a superposition of both a charge nematic that GL free energies will have much of the same form and a nematic-spin-nematic state. On the square lattice, even if the actual coefficients may be different, since we the dx2−y2 distortion pattern is more favorable than that typically need a strong enough exchange interaction in a of dxy. Thus the transition should be Ising-like. In the non s-wave channel. In the following, we will summarize presence of SO coupling, while preserving both parity a number of known experimental systems (and numeri- and TR symmetries, the magnetic field can couple to the cal) which suggest possible directions to search for the α dx2−y2 channel order parameter through terms in the free and β-phases. energy of the form a. 3He : The spin exchange interaction in the Fermi (B2 B2)B n , (11.1) liquid state of 3He is very strong, as exhibited in the x − y z z,1 low frequency paramagnon modes [1]. in this system, which is cubic in B, and the spin fluctuations are known to mediate the p-wave Cooper pairing. The Landau parameter F a in 3He 1 B~ ~na na (11.2) was determined to be negative from various experiments · [63, 64, 65, 66], including the normal-state spin diffusion (where na is the charge nematic order parameter), which constant, spin-wave spectrum, and the temperature de- is linear in B. Thus, an in-plane B field can lock the pendence of the specific heat. It varies from around 0.5 orientation of the nematic-spin-nematic order parameter. − to 1.2 with increasing pressures to the melting point. This effect is more pronounce if the system is in a charge − a Although F1 is not negatively large enough to pass the nematic phase. critical point, we expect that reasonably strong fluctua- d. 2DEG in large magnetic fields : Currently, the tion effects exist. strongest experimental evidence for a charge nematic b. URu2Si2 : The heavy fermion compound (fully polarized) state is in the case of a 2DEG in a URu2Si2 undergoes a phase transition at 17K. The tiny large perpendicular magnetic field[11, 12, 13]. In the antiferromagnetic moment developed in the low tempera- second and higher Landau levels, a huge and strongly ture phase can not explain the large entropy loss. About temperature-dependent resistance anisotropy is seen in 40% density of state density is lost at low temperatures. ultra-high mobility samples, for filling factors near the Currently the low temperature phase is believed to be middle of the partially filled Landau level. In this regime, characterized by an unknown ‘hidden’ order parameter the I-V curves are clearly linear at low bias. No evidence [67]. An important experimental result of nuclear mag- is seen of a threshold voltage or of broad band noise, netic resonance (NMR) [68] shows the broadening of the both of which should be present if the 2DEG would be in line shape below Tc. This implies the appearance of a a stripe state, which is favored by Hartree-Fock calcula- random magnetic field in the hidden ordered phase. Re- tions. Both effects are seen in nearby reentrant integer- cently, Varma et al. proposed the p-wave α-phase as the Hall states. Thus, the simplest interpretation of the ex- hidden ordered phase [31, 32]. They fit reasonably well periments is that the ground state is a polarized charge 23 nematic[9, 10]. There is still a poorly understood alter- neutron scattering. Nevertheless, in the ordered phase, nation effect: the strength of the anisotropic resistance as we have discussed spin-spin correlation function cou- appears to alternate between the fully polarized state and ples to the Goldstone modes, and develops a character- the state with partial polarization. Although this effect istic resonance structure, which should be accessible to could be explained in terms of microscopic calculations inelastic neutron scattering. An experimental detection of the order parameter, it is still possible that the latter of this resonance and its appearance or disappearance in may suggest some form of partially polarized nematic- the ordered or disordered phases can justify the existence spin-nematic order. there are no reliable calculations of of these phases. On the other hand, Fermi surface config- Landau parameters in these compressible phases. urations and single particle spectra in the α and β-phases e. The 2DEG at zero magnetic field : The 2DEG at are different from the normal state Fermi liquids. If the low densities is a strongly coupled system and much work angle resolved photon emission spectroscopy (ARPES) has been done on this system in the context of its appar- experiment can be performed, it can readily tell these ent metal-insulator transition. What interests us is the phases. In the β-phase, the order parameter is similar possibility that this system may have phases of the type to the Rashba SO coupling, the method to detect the discussed here. (The possibility of non-uniform “micro- Rashba coupling can be applied here. For example, from emulsion” phases in the 2DEG was proposed recently by the beat pattern of the Shubnikov-De Hass oscillations Jamei and collaborators[69].) A numerical evaluation to of the ρ(B), we can determine the spin-splitting of two a helicity bands. The asymmetry of the confining potential the Landau parameter F1 in 2D, performed by Kwon et al, [70] by using variational quantum Monte-Carlo, certainly will also contribute some part to the final spin- found that F a is negative and decreasing from 0.19 at orbit coupling. But, when the dynamically generated 1 − rs =1 to 0.27 at rs = 5. On the other hand, Chen et part dominates, it will not sensitive to the asymmetry of al. [71, 72]− investigated the many-body renormalization the confining potential. effect to the Rashba SO coupling due to the exchange a interaction in the F1 channel. They found the renormal- ized SO coupling is amplified significantly at large rs by XII. CONCLUSIONS using a local field approximation. More numerical work to check whether Pomeranchuk instabilities can occur in In summary, we have studied the Pomeranchuk insta- this system would be desirable. bility involving spin in the high orbital partial wave chan- f. Ultra-cold atomic gases with a p-wave Feshbach res- nels. GL free energies are construct to understand the onance : Another type of strongly interacting system is ordered phase patterns after the instabilities take place. cold atoms with Feshbach resonances. Recently, a inter- The ordered phases can be classified into α and β-phases species p-wave Feshbach resonance has been experimen- as an analogy to the superfluid 3He A and B phases. 6 tally studied by using the two component Li atoms [73]. Both phases are characterized by a certain type of effec- In the regime of positive scattering length, close to the tive SO coupling, gives rise a mechanism to generate SO a resonance the Landau parameter of F1 should be neg- couplings in a non-relativistic systems. In the α-phase, ative and large in magnitude. Thus, this system would the Fermi surfaces exhibit an anisotropic distortion, while appear to be a good candidate to observe these phases. those in the β-phase still keep the circular or spherical However, since the p-wave Feshbach resonance is subject shapes undistorted. We further analyze the collective to a large loss-rate of particles, it is not clear whether modes in the ordered phases at the RPA level. Similarly it would be possible to use this approach to observe a to the Pomeranchuk instability in the spin-singlet density stable system with a Pomeranchuk instability near the channel, the density channel Goldstone modes in the α- resonance. phase also shows anisotropic overdamping, except along g. How to detect these phases : We also propose sev- some specific symmetry-determined directions. The spin eral experimental methods to detect the α and β-phases channel Goldstone modes are found to exhibit nearly (see also the discussion in Ref.[35]). For the case of the isotropic linear dispersion relations at small propagating spatially anisotropic α-phases, evidence for strongly tem- wave vectors. The Goldstone modes in the β-phase are perature dependent anisotropy in the transport proper- relative spin-orbit rotation modes with linear dispersion ties (as well as the tunability of this effect by either ex- relation at l 2. The spin-wave modes in both ordered α ternal in-plane magnetic fields and/or uniaxial stress) as and β-phases≥ couples to the Goldstone modes, which thus seen in Sr3Ru2O7 and in the 2DEG can provide direct develop characteristic resonance peaks, that can be ob- evidence for the spacial nematic nature of these phases. served in inelastic neutron scattering experiments. The Spatially nematic phases exhibit anisotropic transport p-wave channel is special in that the β-phase can develop properties even in a single-domain sample[4]. More dif- a spontaneous chiral Lifshitz instability in the originally ficult is to determine their spin structure. Because no nonchiral systems. The GL analysis was performed to ob- magnetic moments appear in both phases, elastic neu- tain the twist pattern in the ground state. We also review tron scattering does not exhibit the regular Bragg peaks. the current experiment status for searching these instabil- Since the Goldstone modes are combined spin and orbital ities in various systems, including 3He, the heavy fermion excitations, they can not be directly measured through compound URu2Si2, the bilayer ruthenate Sr3Ru2O7, 2D 24 electron gases, and p-wave Feshbach resonances with cold microscopic two-body SU(2) invariant interaction fermionic atoms. The Sr3Ru2O7 seems the most promis- ing systems to exhibit such an instability in the 2D d- V (~r1, ~r2)= Vc(~r1 ~r2)+ Vs(~r1 ~r2)S~1 S~2, (A2) wave channel. However, investigation on the SO cou- − − · pling effect is needed to understand the suppression of where the Vc and Vs are the spin-independent and de- the resistivity anomaly due to the in-plane field. pendent parts, respectively. At the Hartree-Fock level, There are still many important properties of the spin f s,a(p,p′) are triplet Pomeranchuk instabilities yet to be explored. s ′ 1 ′ 3 ′ In the paper, we did not discuss the behavior of the f (~p, ~p ) = Vc(0) Vc(~p ~p ) Vs(~p ~p ), fermionic degrees of freedom, which are expected to − 2 − − 8 − a ′ 1 ′ 1 1 ′ be strongly anomalous. Generally speaking, the over- f (~p, ~p ) = Vc(~p ~p )+ Vs(0) + Vs(~p ~p ). damped density channel Goldstone modes in the α-phase −2 − 4 8 − strongly couple to the fermions, which is expected to lead (A3) to a non-Fermi liquid behavior as in the case of the den- s,a ′ sity channel Pomeranchuk instabilities [4, 17]. However, f (~p, ~p ) can be further decomposed into different or- the Goldstone modes in the β-phase is not damped as bital angular momentum channels as l 2, thus similarly to the case of itinerant ferromag- 1 s,a ′ ′ ≥ s,a d cos θ f (ˆp pˆ )P (ˆp pˆ ) in 3D, nets, the β-phase remains a Fermi liquid. The p-wave f = −1 · l · (A4) channel in particularly interesting. We have shown in l dφ f s,a(ˆp pˆ′)cos lφ in 2D ( R 2π · Eq. (2.12), that p-wave paramagnon fluctuations couple R to fermions as an SU(2) gauge field. The linear deriva- where Pl is the lth-order Legendre polynomial. For each s,a tive terms in the G-L free energy also become relevant in channel of Fl , Landau-Pomeranchuk (LP) instability the finite temperature non-Gaussian regime. The Hertz- [3] occurs at a Mills type critical theory for the Fl contains new features compared to the ferromagnetic ones. We defer to a future s,a s,a < (2l +1) in3D, Fl = N(0)fl − (A5) < (2 δl,0) in 2D, publication to address the above interesting questions. − − where N(0) is the density of states at the Fermi energy. Acknowledgments

APPENDIX B: THE GOLDSTONE MODES OF We thank L. Balents, M. Beasley, S. L. Cooper, S. Das THE SPIN OSCILLATION IN THE ALPHA Sarma, M. P. A. Fisher, S. A. Grigera, S. A. Kivelson, H. PHASE Y. Kee, M. J. Lawler, A. W. W. Ludwig, A. P. Mackenzie, C. M. Varma, and J. Zaanen for helpful discussions. CW In this section, we calculate the spin channel Goldstone thanks A. L. Fetter and Y. B. Kim for their pointing out modes in the 2D α-phases at small wavevector v q/n¯ 1 the Lifshitz instability in the β-phase. This work was F and low frequency ω/n¯ 1. The expression for≪ the supported in part by the National Science Foundation dispersion of L (~q, ω)≪ is through the grants PHY99-07949 at the Kavli Institute x±iy,1 for Theoretical Physics (C.W.), DMR-04-42537 at the 1 d2k University of Illinois (E.F.), DMR-0342832 at Stanford L (~q, ω)= κq2 + +2 cos2 lθ x+iy,1 f a (2π)2 k University (S.C.Z.), and by the Department of Energy, | 1 | Z Oﬃce of Basic Energy Sciences, under the contracts DE- n [ξ (~k ~q )] n [ξ (~k + ~q )] f ↓ − 2 − f ↑ 2 FG02-91ER45439 at the Frederick Seitz Materials Re- ~q ~q , × ω + iη + ξ (~k ) ξ (~k + ) search Laboratory of the University of Illinois (E.F. and ↓ − 2 − ↑ 2 K.S.), and DE-AC03-76SF00515 at Stanford University (B1) (S.C.Z.). where ξ↑,↓(k)= ǫ(k) µ n¯ cos lθk. Following the procedure in Ref. [4], we separate− ∓ Eq. (B1) into a static part APPENDIX A: LANDAU INTERACTION Lx+iy,1(~q, ω) and a dynamic part Mx+iy,1(~q, ω) as PARAMETERS Lx+iy,1(~q, ω)= Lx+iy,1(~q, 0)+ Mx+iy,1(~q, ω). (B2)

Landau-Fermi liquid theory is characterized by the in- At ~q = 0,ω = 0, from the self-consistent equation Eq. teraction functions which describe the forward scattering 1 (4.3), the integral cancels the constant term a as re- process between quasi-particles as |f1 | quired by Goldstone theorem. The detailed form of the ′ s ′ a ′ static part at small but nonzero ~q is difficult to evalu- fαβ,γδ(~p, ~p )= f (~p, ~p )+ f (~p, ~p )~σαβ ~σγδ, (A1) · ate due to anisotropic Fermi surfaces. Because of the where ~p and ~p′ lie close to the Fermi surface. The ex- breaking of parity in the α-phase, it seems that the lead- pressions of f s and f a can be obtained through a general ing order contribution should be linear to q. However, 25 from the Ginzburg-Landau analysis in Sec. VII A, the To evaluate this integral, we make several simplifications: linear derivative term in Eq. (3.5) does not contribute the non-linear part in ǫ(~k) is neglected and the linear to the coupling among Goldstone modes, i.e., the uni- order in ~q is kept in the denominator. We arrive at form ground state is stable in contrast to the case in the β-phase. As a result, the dependence on ~q should start from the quadratic order, bringing a correction to the coefficient κ. For simplicity, we neglect this correction for it does not cause qualitatively different result. Thus, we arrive at

2 Lx+iy,1(~q, ω =0)= κq . (B3)

The dynamic part, Mx+iy,1(~q, ω), can be expressed as

2 2 d k cos lθk ω Mx+iy,1 = 2 − (2π)2 ω + iη + ξ (~k ~q ) ξ (~k + ~q ) Z ↓ − 2 − ↑ 2 ~q ~q nf [ξ↓(~k )] nf [ξ↑(~k + )] − 2 − 2 . (B4) × ξ (~k ~q ) ξ (~k + ~q ) ↓ − 2 − ↑ 2

dθ cos2 lθ ω kdk ~q ~q k k − (n [ξ (~k )] n [ξ (~k + )]), 2π ω + iη +2¯n cos lθ qv cos(θ φ) 2¯n cos lθ qv cos(θ φ) 2π f ↓ − 2 − f ↑ 2 Z k − F k − k − F k − Z dθ ω cos2 lθ k2 k2 = k − k 2 − 1 , 2π ω + iη +2¯n cos lθ qv cos(θ φ) 2π[2¯n cos lθ qv cos(θ φ)] Z k − F k − k − F k − (B5)

where k1 and k2 satisfy nf [ξ↑(~k1 + ~q/2)] = 0 and over z = exp(iθk) and deﬁne the density of the states at n [ξ (~k ~q/2)] = 0 respectively, and φ is the azimuthal chemical potential in the ordered state as f ↓ 2 − angle of ~q. k1 and k2 can be approximated by

2 2 x n¯ cos lθk q kF (1 x /4) k1,2 = kF (1 ( cos(θk φ)) N<(0) = − , (B7) − 4 ± vF − 2 − vF π + O(q2). (B6)

We now transfer the integral over θk to an integral The integral above now reads as:

l −l N<(0) dz z + z 2 ω ( ) l −l 2πi z 2 ω + iη +2¯n z +z qvF (ze−iφ + z−1eiφ)+ O(q2) I 2 − 2 l −l 2 N<(0)ω (z + z ) Res l −l −iφ −1 iφ . (B8) ≈ 4 z [2ω +2iη +2¯n(z + z ) qvF (ze + z e )] |Xz|<1 −

This integral can be calculated by evaluating the residues equation: at poles inside the unit circle. There is one pole at 0, 2ω +2iη +2¯n(zl + z−l) qv (ze−iφ + z−1eiφ)=0. one pole at , and 2l poles, from the solutions of the − F ∞ (B9) The pole at z = is not inside the unit circle and does not contribute to∞ the integral. 26

The pole at z = 0 has diﬀerent behavior for diﬀer- most only gives negligible higher order terms. In the limit 2ω ent values of l. The residue is iφ 2 for l = 1, of small q and ω, Eq. (B9) can be solved perturbatively (2¯n−e qvF ) 2 2 2 − i(2m−1)π 4¯nω−e iφq v F ω as a power series of q and ω as zm = exp( 2l )+ 8¯n3 for l = 2, and 2¯n2 for all l> 2. To lead- − − ω O(q)+ O(ω), where m = 1, 2,..., 2l. To the leading or- ing order, all the residues become 2¯n2 at l 1. Next we discuss the poles at the solutions− of Eq. (B9).≥ For l 1, der, this type of poles are all simple poles, and the residue ≥ of 1/ 2ω +2iη +2¯n(zl + z−l) qv (ze−iφ + z−1eiφ) is not all of these poles are located inside the unit circle. − F at the order of O( 1 ). Therefore, the contribution from However, we will not bother to tell which poles are inside 2¯n the unit circle because we can show that these poles at the pole zm is:

l −l 2 N<(0)ω 1 (z + z ) Res( l −l −iφ −1 iφ )z=zm 4 z 2ω +2iη +2¯n(z + z ) qvF (ze + z e ) −iφ −1−iφ 2 2 N<(0)ω (2ω +2iη qvF (zme + zm e )) 1 ω(ω + q) − 2 l−1 O( 3 ), (B10) ≈ 2 zm(2¯n) 4lnz¯ m ≈ n¯

which is negligible to order of O(ω2/n¯2) and O(q2/n¯2). at l 1. In short, for l 1, only the pole at z = 0 contributes ≥ to the integral. The≥ result of the ﬂuctuation kernel is

N (0) L (~q, ω)= κq2 < ω2. (B11) x+iy,1 − 4¯n2

From the self-consistent equation we ﬁnd that N<(0) = 2 a . Therefore, the spin channel Goldstone mode reads |fl |

ω2 L (~q, ω) = κq2 x+iy,1 − 2¯n2 f a(0) | l | N(0) ω2 = κq2 , (B12) − 2 F a n¯2 | l |

[1] A. J. Leggett, Rev. Mod. Phys. 47, 331 (1975). [12] K. B. Cooper, M. P. Lilly, J. P. Eisenstein, L. N. Pfeiffer, [2] G. Baym and C. Pethick, Landau Fermi-liquid theory and K. W. West, Phys. Rev. B 65, 241313 (2002). (John Wiley & Sons, Inc., 1984). [13] R. R. Du, D. C. Tsui, H. L. Stormer, L. N. Pfeiffer, K. W. [3] I. I. Pomeranchuk, Soviet Physics JETP 8, 361 (1959). Baldwin, and K. W. West, Sol. Stat. Comm. 109, 389 [4] V. Oganesyan, S. A. Kivelson, and E. Fradkin, Phys. (1999). Rev. B 64, 195109 (2001). [14] S. A. Grigera, P. Gegenwart, R. A. Borzi, F. Weickert, [5] P. G. de Gennnes and J. Prost, The Physics of Liquid A. J. Schofield, R. S. Perry, T. Tayama, T. Sakakibara, Crystals (Clarendon Press, Oxford, UK, 1993), 2nd ed. Y. Maeno, A. G. Green, et al., Science 306, 1154 (2004). [6] P. M. Chaikin and T. C. Lubensky, Principles of Con- [15] A. P. Mackenzie and S. A. Grigera, private communica- densed Matter Physics (Cambridge University Press, tion. 1995). [16] D. G. Barci and L. E. Oxman, Phys. Rev. B 67, 205108 [7] S. A. Kivelson, E. Fradkin, and V. J. Emery, Nature 393, (2003). 550 (1998). [17] M. J. Lawler, D. G. Barci, V. Fernandez, E. Fradkin, and [8] S. A. Kivelson, I. P. Bindloss, E. Fradkin, V. Oganesyan, L. Oxman, Phys. Rev. B 73, 085101 (2006). J. M. Tranquada, A. Kapitulnik, and C. Howald, Rev. [18] J. Nilsson and A. H. Castro Neto, Phys. Rev. B 72, Mod. Phys. 75, 1201 (2003). 195104 (2005). [9] E. Fradkin and S. A. Kivelson, Phys. Rev. B 59, 8065 [19] J. Quintanilla and A. J. Schofield, Phys. Rev. B 74, (1999). 115126 (2006). [10] E. Fradkin, S. A. Kivelson, E. Manousakis, and K. Nho, [20] C. J. Halboth and W. Metzner, Phys. Rev. Lett. 85, 5162 Phy. Rev. Lett. 84, 1982 (2000). (2000). [11] M. P. Lilly, K. B. Cooper, J. P. Eisenstein, L. N. Pfeiffer, [21] L. Dell’Anna and W. Metzner, Phys. Rev. B 73, 045127 and K. W. West, Phy. Rev. Lett. 82, 394 (1999). (2006). 27

[22] H. Y. Kee, Phys. Rev. B 67, 73105 (2003). [47] H. Y. Kee, E. H. Kim, and C. H. Chung, Phys. Rev. B [23] I. Khakvine, C. H. Chung, V. Oganesyan, and H. Y. Kee, 68, 245109 (2003). Phys. Rev. B 70, 155110 (2004). [48] H. Doh, N. Friedman, and H.-Y. Kee, Phys. Rev. B. 72, [24] H. Yamase, V. Oganesyan, and W. Metzner, Phys. Rev. 125117 (2006). B 72, 35114 (2005). [49] R. Winkler, Phys. Rev. B 62, 4245 (2000). [25] S. A. Kivelson, E. Fradkin, and T. H. Geballe, Phys. Rev. [50] J. Schliemann and D. Loss, Phys. Rev. B 71, 085308 B 69, 144505 (2004). (2005). [26] M. J. Lawler and E. Fradkin (2006), unpublished; [51] P. McGraw, Phy. Rev. D 50, 952 (2004). cond-mat/0605203. [52] F. Zhou, Int. J. Mod. Phys. 17, 2643 (2003). [27] J. E. Hirsch, Phys. Rev. B 41, 6820 (1990). [53] C. Wu, J. P. Hu, and S. C. Zhang, cond-mat/0512602 [28] J. E. Hirsch, Phys. Rev. B 41, 6828 (1990). (unpublised). [29] L. P. Gorkov and A. Sokol, Phys. Rev. Lett. 69, 2586 [54] A. S. Schwarz, Nucl. Phys. B 208, 141 (1982). (1999). [55] M. Bucher, H. K. Lo, and J. Preskill, Nucl. Phys. B 386, [30] C. J. Wu and S. C. Zhang, Phys. Rev. Lett. 93, 36403 3 (1999). (2004). [56] D. Belitz, T. R. Kirkpatrick, and T. Vojta, Rev. Mod. [31] C. M. Varma, Philos. Mag. 85, 1657 (2005). Phys 77, 579 (2005). [32] C. M. Varma and L. J. Zhu, Phys. Rev. Lett. 96, 036405 [57] H. v. Lvhneysen, A. Rosch, M. Vojta, and P. Wolfle, (2006). cond-mat/0606317 (unpublished). [33] H. Y. Kee and E. H. Kim, Phys. Rev. B 71, 184402 [58] E. Demler, W. Hanke, and S.-C. Zhang, Rev. Mod. Phys. (2005). 76, 909 (2004). [34] C. Honerkamp, Phys. Rev. B 72, 115103 (2005). [59] A. L. Fetter, private communication. [35] D. Podoslky and E. Demler, New J. Phys. 7, 59 (2005). [60] Y. B. Kim, private communication. [36] J. Zaanen and Z. Nussinov, Phys. Status Solidi B 236, [61] T. C. Lubensky and L. Radzihovsky, Phys. Rev. E 66, 332 (2003). 31704 (2002). [37] In this paper we consider only the effects of charge and [62] A. P. Mackenzie and Y. Maeno, Rev. Mod. Phys. 75, 657 spin degrees of freedom. Thus, ‘orbital’ effects refer here (2003). only to channels with non-zero orbital angular momen- [63] A. J. Leggett, J. Phys. C 3, 448 (1970). tum l. These are not the “orbital” degrees of freedom of [64] L. R. Corruccini, D. D. Osheroff, D. M. Lee, and R. C. multiband models commonly used in ruthenate and man- Richards, Phys. Rev. Lett. 27, 650 (1971). ganite materials. Throughout this paper we consider sys- [65] D. D. Osheroff, Physica B & C 90, 20 (1977). tems in which the conduction electrons correspond only [66] D. S. Greywall, Phys. Rev. B 27, 2747 (1983). to one effective band. [67] V. Tripathi, P. Chandra, and P. Coleman, J. Phys. : [38] B. A. Bernevig, J. Orenstein, and S. C. Zhang (2006), Cond. Matt. 17, 5285 (2005). unpublished; cond-mat/0606196. [68] O. O. Bernal, C. Rodrigues, A. Martinez, H. G. Lukefahr, [39] S. Murakami, N. Nagaosa, and S. C. Zhang, Science 301, D. E. MacLaughlin, A. A. Menovsky, and J. A. Mydosh, 1348 (2003). Phys. Rev. Lett. 87, 196402 (2001). [40] J. Sinova, D. Culcer, Q. Niu, N. A. Sinitsyn, T. Jung- [69] R. Jamei, S. A. Kivelson, and B. Spivak, Phy. Rev. Lett. wirth, and A. H. MacDonald, Phys. Rev. Lett. 92, 126603 94, 056805 (2005). (2004). [70] Y. Kwon, D. M. Ceperley, and R. M. Martin, Phys. Rev. [41] Y. K. Kato, R. C. Myers, A. C. Gossard, and D. D. B 50, 1684 (1994). Awschalom, Phys. Rev. Lett. 93, 176601 (2004). [71] G. H. Chen and M. E. Raikh, Phys. Rev. B 60, 4826 [42] J. Wunderlich, B. Kaestner, J. Sinova, and T. Jungwirth, (1999). Phys. Rev. Lett. 94, 47204 (2005). [72] G.-H. Chen, M. E. Raikh, and Y.-S. Wu, Phys. Rev. B [43] B. Binz, A. Vishwanath, and V. Aji, Phys. Rev. Lett. 96, 61, R10539 (2000). 207202 (2006). [73] J. Zhang, E. G. M. van Kempen, T. Bourdel, [44] J. A. Hertz, Phys. Rev. B 14, 1165 (1976). L. Khaykovich, J. Cubizolles, F. Chevy, M. Teichmann, [45] A. J. Millis, Phys. Rev. B. 48, 7183 (1993). L. Tarruell, S. J. J. M. F. Kokkelmans, and C. Salomon, [46] D. Vollhardt and P. Wolfle, The superfluid phases of he- Phys. Rev. A 70, 30702(R) (2004). lium 3 (Taylor & Francis, 1990). w

0 q Q