ISOTROPY GROUPS OF THE ACTION OF ORTHOGONAL SIMILARITY ON SYMMETRIC MATRICES TADEJ STARCIˇ Cˇ Abstract. We find an algorithmic procedure that enables to compute and to de- scribe the structure of the isotropy subgroups of the group of complex orthogonal matrices with respect to the action of similarity on complex symmetric matrices. A key step in our proof is to solve a certain rectangular block upper-triangular Toeplitz matrix equation. 1. Introduction and the main result All matrices considered in this paper are complex unless otherwise is stated. Cm n C We use the notation × for the set of matrices of size m n. By Sn( ) we denote the vector space of all n n symmetric matrices; A is symmetric× if and only if T C× A = A . Let further On( ) be the subgroup of orthogonal matrices in the group C of nonsingular n n matrices GLn( ). A matrix Q is orthogonal if and only if T 1 × C Q = (Q )− . The action of orthogonal similarity on Sn( ) is defined as follows: (1.1) Φ : O (C) S (C) S (C), (Q,A) QT AQ. n × n → n 7→ The isotropy group at A S (C) with respect to the action (1.1) is ∈ n (1.2) Σ := Q O (C) QT AQ = A A { ∈ n | } and the orbit of A is (1.3) Orb(A) := QT AQ Q O (C) . { | ∈ n } An orbit thus consists of orthogonally similar matrices and the isotropy groups of these matrices are isomorphic. C The action (1.1) describes symmetries of Sn( ). Hua’s fundamental results [10, 11, 12] on the geometry of symmetric matrices assure that the study of sym- metric matrices under T -congruence (which includes (1.1)) is quite general. An arXiv:2108.06757v1 [math.DG] 15 Aug 2021 important information concerning a group action is provided by its orbits and the corresponding isotropy groups (see monographs [7, 15]), and to find these for the action (1.1) is the main purpose of this paper. Moreover, the so-called linear isotropy representation at A S (C) is the restricion of (1.1): ∈ n T T T n n (1.4) Σ T T , (Q,A) Q AQ, T := X A + AX X = X C × , A × A → A 7→ A { | − ∈ } C a representation of ΣA on a complex vector space TA Sn( ) associated to the C ⊂ tangent space of Orb(A) Sn( ) at A (see also Sec. 4). It is closely related to in- variant objects of Orb(A)⊂ (see [7, 13]). On the other hand (1.1) can be seen as a Date: August 13, 2021. 2000 Mathematics Subject Classification. 15A24, 51H30, 32M05 . Key words and phrases. isotropy groups, matrix equations, orthogonal matrices, symmetric matri- ces, unipotent group, Toeplitz matrices Research supported by grant P1-0291 from ARRS, Republic of Slovenia. 1 2 TADEJ STARCIˇ Cˇ C representation of On( ); note that the classification of representations of complex classical groups along with their invariants is well understood (see e.g. [18]). Fi- nally, the isotropy groups of (1.1) are interesting from the linear algebraic point of view (check Remark 1.4). To be able to compute the isotropy groups, it is essential to have simple repre- sentatives of orbits. Thus we recall the symmetric canonical form under similarity; remember that symmetric matrices are similar if and only if they are orthogonally similar (see e.g. [6]). Given a matrix A with its Jordan canonical form: (1.5) J(A) = J (λ ), λ C, nj j j ∈ Mj where z 1 0 . z .. ( ) := C ( -by- ) Jn z , z n n , .. ∈ . 1 0 z the symmetric canonical form is (1.6) S(A) = Knj (λj), Mj in which 2z 1 0 0 1 0 − 1 .. .. .. .. (1.7) ( ) := 1 . + . 1 C ( -by- ) Kn z . i . , z n n . 2 .. .. 1 1 .. .. ∈ 0 1 2z −0 1 0 It is uniquely determined up to a permutation of its direct summands. See [5] for the tridiagonal symmetric canonical form. Since the equation QT AQ = A is equivalent to (J(A))X = X(J(A)) with J(A) = 1 1 PAP − , X = PQP − , the following fact on isotropy groups follows immediately from the classical result on solutions of Sylvester’s equation (see Theorem 2.1 (1)). k Proposition 1.1. If λ1,...,λk are distinct eigenvalues of S = j=1 Sj, where each Sj is a direct sum whoose summands are of the form (1.7) and correspL ond to the eigenvalue Σ = k Σ = λj, it then follows that S j=1 Sj . Furthermore, if Sj λj Inj for some index j, then Σ = O (C). (We denoteL the n n identity-matrix by I .) Sj nj × n Therefore the isotropy groups under (1.1) of matrices with all distinct eigenval- ues (hence with nonvanishing discriminants of their characteristic polynomials) are trivial. The situation in the generic case (on a complement of a complex ana- lytic subset of codimension 1) is thus quite simple. Our aim is to inspect the nongeneric matrices (especially nondiagonalizable). The principal object of the investigation will be (up to similarity) the group of all nonsingular matrices commuting with a given square matrix M, i.e. nonsingular solutions of the homogeneous linear Sylvester’s equation MX = XM; see Sec. 2 for 3 its properties. First, recall that a block upper-triangular Toeplitz matrix is: A0 A1 A2 ... ... Aβ 1 .− 0 . A0 A1 A2 . . .. .. . A0 A1 . T (A0,A1,...,Aβ 1)= (β-by-β), − . . .. .. .. A 2 . . .. .. . A1 0 ... ... ... 0 A0 Cm n β where A0,A1,...,Aβ 1 × and T (A0,A1,...,Aβ 1) = [Tjk] =1 with Tjk = 0 for − ∈ − j,k j>k and T = T . Next, suppose α >α > >α and m ,...,m N. jk (j+1)(k+1) 1 2 ··· N 1 N ∈ Let be an N N block matrix such that its block rs is a rectangle αr αs block upper-triangularX × Toeplitz matrix with blocks od sizeXm m : × r × s [0 ], α <α Trs r s (1.8) = [ ]N , = rs , α >α , b = min α ,α , rs r,s=1 rs T0 r s rs s r X X X " # { } , α = α Trs r s in which rs is a brs brs block upper-triangular Toeplitz matrix. It turns out that orthogonalT solutions× of the equation SX = XS with S of the form (1.6) are related to matrices of the form (1.8) such that the following properties are satisfied: (I) The nonzero entries of for r>s can be taken as free variables. Xrs r r r C (II) If s = r, then rr = T (A ,...,A ), where A Om ( ) can be any orthogonal X 0 αr 1 0 ∈ r matrix, and for α 2, j 1,...,α − 1 we have Ar = Ar Zr + Dr for some freely r ≥ ∈{ r − } j 0 j j r r T r Cmr mr chosen skew-symmetric Zj = (Zj ) of size mr mr , and with Dj × de- − × p p ∈ pending uniquely (and polynomially) on the entries of A , Z with j 1,...,j 1 , 0 j ∈{ − } p 1,...,N and on the entries of for p,t 1,...,N with p>t. ∈{ } Xpt ∈{ } e e (III) The entries of rs for r<s are uniquely determined (the dependence is poly- nomial) by the entriesX of for p,t 1,...,N with p t. Xpt ∈{ } ≥ A simple example of a block diagonal matrix of the form (1.8) is N r r (1.9) = T (Imr ,W1 ,...,Wα 1), W r − Mr=1 n r 1 r r 1 r r T r W1 := Z1, Wn+1 := Zn+1 (Wj ) Wn j+1 , n 1, 2 2 − − ≥ Xj=1 r in which all Zn are skew-symmetric. Another special matrix of the form (1.8) contains the identity matrix as principal submatrix, formed by all blocks except those at the p-th and the t-th columns and rows, while blocks in the p-th and the t-th columns and rows are as follows: [ 0 rs ], α <α U r s k k N k rs (1.10) p,t(F) = [( p,t(F))rs] =1, ( p,t(F))rs = U , αr >αs , p<t, G G r,s G 0 h ,i α = α Urs r s 4 TADEJ STARCIˇ Cˇ where αr I , r = s, j=1 mr 1 rr rr rs = ⊕ , r,s p,t , rr = T (Imr ,A1 ,...,Aα 1), r p,t , U ( 0, r , s { } { } U r − ∈{ } T n pp an 1(F F) , j = n(2k + α β) 1 1 2n A = − − , an = , j ( 0, otherwise −22n+1 n + 1 n ! T n tt an 1(FF ) , j = n(2k + α β) A = − − , j ( 0, otherwise = N k (F), = N k ( FT ), 0 k α 1, Upt αt Utp αt − ≤ ≤ t − k m m in which N (F)isa β β block matrix with F C p× t on the k-th diagonal above β × ∈ the main diagonal for k 1 (on the main diagonal for k = 0) and zeros othervise. ≥ Example 1.2. N = 3, α = 4, α = 2, α = 1, m = 2, m = 3, m = 1; F C2 3: 1 2 3 1 2 3 ∈ × 1 T T I2 0 2 F F 0 F 0 0 − 1 T − T 0 I2 0 2 F F 0 F 0 − − 0 0 I2 0 0 0 0 0 (F)= 00 0 I2 0 0 0 G1,2 0 0 F 0 I 0 0 3 00 0 F 0 I3 0 00 0 0 0 0 1 Our main result is the following.