Submitted exclusively to the London Mathematical Society doi:10.1112/0000/000000 Schur–Weyl duality for orthogonal groups Stephen Doty and Jun Hu Abstract We prove Schur–Weyl duality between the Brauer algebra Bn(m) and the orthogonal group Om(K) over an arbitrary infinite field K of odd characteristic. If m is even, we show that each connected component of the orthogonal monoid is a normal variety; this implies that the orthogonal Schur algebra associated to the identity component is a generalized Schur algebra. As an application of the main result, an explicit and characteristic-free description of the annihilator ⊗n of n-tensor space V in the Brauer algebra Bn(m) is also given. 1. Introduction Let m,n N. Write λ n to mean that λ = (λ , λ ,... ) is a partition of n, and denote by ∈ ⊢ 1 2 ℓ(λ) the largest integer i such that λi = 0. Let K be an infinite field and V an6 m-dimensional K-vector space. The natural left action of the general linear group GL(V ) on V ⊗n commutes with the right permutation action of the symmetric group Sn. Let ϕ, ψ be the corresponding natural representations ϕ : (KS )op End V ⊗n , ψ : KGL(V ) End V ⊗n , n → K → K respectively. The well-known Schur–Weyl duality (see [8], [17], [37], [46 ], [47 ]) says that op ⊗n (a) ϕ (KSn) = EndKGL(V ) V , and if m n then ϕ is injective, and hence an ⊗n ≥ isomorphism onto EndKGL(V ) V , ⊗n (b) ψ KGL(V ) = End S V , K n (c) if char K = 0, then there is an irreducible KGL(V )-KS -bimodule decomposition n V ⊗n = ∆ Sλ, λ ⊗ λ=(λ1 ,λ2,··· )⊢n ℓ(Mλ)≤m λ where ∆λ (resp., S ) denotes the irreducible KGL(V )-module (resp., irreducible KSn- module) associated to λ. There are also Schur–Weyl dualities for symplectic groups and orthogonal groups in the arXiv:0712.0944v3 [math.RT] 8 Oct 2008 semisimple case, i.e., when K has characteristic zero; see [3], [4] and [5]. In these cases, the symmetric group will be replaced by certain specialized Brauer algebras. We are mostly interested in the non-semisimple case. In [10], Schur–Weyl duality between the Brauer algebra Bn( 2m) and the symplectic group Sp2m(K) over an arbitrary infinite field K was proved. In [30],− the second author gave an explicit and characteristic-free description of the annihilator of n-tensor space V ⊗n in the Brauer algebra B ( 2m). n − The aim of this work is to generalize these results to the orthogonal case. We first recall the definition of orthogonal group over an arbitrary infinite field K with char K = 2. Let V be an 6 2000 Mathematics Subject Classification 20G05 (primary), 20C20 (secondary).. The second author was supported by the National Natural Science Foundation of China (Project 10771014), the Program NCET and the Scientific Research Foundation for the Returned Overseas Chinese Scholars, State Education Ministry. Page 2 of 35 STEPHEN DOTY AND JUN HU m-dimensional K-vector space with a non-degenerate symmetric bilinear form ( , ). Then the orthogonal similitude group (resp., orthogonal group) relative to ( , ) is 0 = d K, such that (gv,gw)= d(v, w) GO(V ) := g GL(V ) ∃ 6 ∈ ∈ v, w V ∀ ∈ resp., O(V ) := g GL(V ) (gv,gw ) = (v, w), v, w V . ∈ ∀ ∈ n o By restriction from GL(V ), we get natural left actions of GO(V ) and O(V ) on V ⊗n. Note that if 0 = d K is such that (gv,gw)= d(v, w) for any v, w V , then (√d−1g)v, (√d−1g)w = 6 ∈ ∈ (v, w) for any v, w V . Therefore, if K is large enough such that √d K for any d K, then g GO(V ) implies∈ that (a id )g O(V ) for some 0 = a K. In that∈ case, ∈ ∈ V ∈ 6 ∈ −1 −1 ψ(g)= ψ (a idV )(a idV )g = ψ a idV ψ (a idV )g −n −n = a idV ⊗n ψ (a idV)g =a ψ (a idV )g . It follows that ψ KO(V ) = ψ KGO(V ) provided K is closed under square roots. We now recall the definition of Brauer algebra. Let x be an indeterminate over Z. The Brauer algebra Bn(x) over Z[x] is a unital Z[x]-algebra with generators s1, ,sn−1,e1, ,en−1 and relations (see [21]): ··· ··· 2 2 si =1, ei = xei, eisi = ei = siei, 1 i n 1, s s = s s , s e = e s , e e = e e , 1∀ i≤ < j≤ 1− n 2, i j j i i j j i i j j i ∀ ≤ − ≤ − sisi+1si = si+1sisi+1, eiei+1ei = ei, ei+1eiei+1 = ei+1, 1 i n 2, s e e = s e , e e s = e s , 1 i ∀ n≤ 2≤. − i i+1 i i+1 i i+1 i i+1 i+1 i ∀ ≤ ≤ − B (x) is a free Z[x]-module with rank (2n 1) (2n 3) 3 1. For any commutative Z[x]- n − · − ··· · algebra R with x specialized to δ R, we define Bn(δ)R := R Z[x] Bn(x). This algebra was first introduced by Richard Brauer∈ (see [3]) in order to describe⊗ how the n-tensor space V ⊗n decomposes into irreducible modules over the orthogonal group O(V ) or the symplectic group Sp(V ), where V is an orthogonal or symplectic vector space. In Brauer’s original formulation, the algebra Bn(x) was defined as the complex linear space with basis the set Bdn of all Brauer n-diagrams, graphs on 2n vertices and n edges with the property that every vertex is incident to precisely one edge. The multiplication of two Brauer n-diagrams is defined using natural concatenation of diagrams. For more details, we refer the readers to [26] and [30]. Note that the subalgebra of B (x) generated by s ,s , ,s is isomorphic to the group algebra of n 1 2 ··· n−1 the symmetric group Sn over Z[x]. The Brauer algebra has been studied in a number of references, e.g., [3], [4], [5], [7], [10], [17], [22], [23], [24], [30], [31], [32], [34], [48]. To set up a Schur–Weyl duality for orthogonal groups, we only need certain specialized Brauer algebras which we now recall. Let Bn(m) := Z Z B (x), where Z is regarded as Z[x]-algebra by specifying x to m. Let B (m) := ⊗ [x] n n K K Z Bn(m), where K is regarded as Z-algebra in the natural way. Then there is a right ⊗ ⊗n action of the specialized Brauer algebra Bn(m)K on the n-tensor space V which commutes with the natural left action of GO(V ). We recall the definition of this action. Let δi,j denote the value of the usual Kronecker delta. For any integer i with 1 i m, we set i′ = m +1 i. We fix an ordered basis v , v , , v of V such that ≤ ≤ − 1 2 ··· m (v , v )= δ ′ , 1 i, j m. i j i,j ∀ ≤ ≤ SCHUR–WEYL DUALITY Page 3 of 35 ⊗n The right action of Bn(m) on V is defined on generators by (vi1 vin )sj := vi1 vij−1 vij+1 vij vij+2 vin , ⊗···⊗ ⊗···⊗ ⊗ ⊗ m⊗ ⊗···⊗ ′ ′ (vi vi )ej := δi ,i vi vi − vk vk vi 1 ⊗···⊗ n j j+1 1 ⊗···⊗ j 1 ⊗ ⊗ ⊗ j+2 Xk=1 v . ⊗···⊗ in That is, the action of sj is by place permutation and the action of ej is by a composition of Weyl’s “contraction” operator with an “expansion” operator. Let ϕ be the K-algebra homomorphism ϕ : (B (m))op End V ⊗n n → K induced by the above action. Lemma 1.1. ([3], [4], [5]) 1) The natural left action of GO(V ) on V ⊗n commutes with the right action of Bn(m). Moreover, if K = C, then op ⊗n ⊗n ϕ Bn(m)C = EndCGO(V ) V = EndCO(V ) V , ψ GO(V ) = ψ O(V ) = End V ⊗n , C C Bn(m)C ⊗n 2) if K = C and m n then ϕis injective, and hence an isomorphism onto EndC V , ≥ GO(V ) 3) if K = C, then there is an irreducible CGO(V )-B (m)C-bimodule decomposition n [n/2] V ⊗n = ∆(λ) D(λ), ⊗ f=0 λ⊢n−2f M ′ M′ λ1+λ2≤m where ∆(λ) (respectively, D(λ)) denotes the irreducible CGO(V )-module (respectively, the ′ ′ ′ irreducible Bn(m)-module) corresponding to λ, and λ = (λ1, λ2, ) denotes the conjugate partition of λ. ··· The first main result in this work removes the restriction on K in part 1) and part 2) of the above theorem. We have Theorem 1.2. For any infinite field K of odd characteristic, we have ⊗n (a) ψ KGO(V ) = EndBn(m) V ; (b) ϕ B (m) = End V ⊗n = End V ⊗n , and if m n, then ϕ is also n KGO(V ) KO(V ) injective, and hence an isomorphism onto ≥ ⊗n EndKGO(V ) V . We remark that the first statement in part b) first appeared in [17] based on a completely different approach. The algebra o ⊗n SK (m,n) := EndBn(m) V is called the orthogonal Schur algebra associated to GO (V). Note that we use a different o definition of orthogonal Schur algebra in Section 2 by defining SK (m,n) to be the linear dual of a certain coalgebra; the two definitions are reconciled in (2.3). Let = Z[1/2].