MATHEMATICS OF COMPUTATION Volume 72, Number 243, Pages 1329{1348 S 0025-5718(03)01505-9 Article electronically published on February 3, 2003 THESPECTRAOFLARGETOEPLITZBANDMATRICES WITH A RANDOMLY PERTURBED ENTRY A. BOTTCHER,¨ M. EMBREE, AND V. I. SOKOLOV Abstract. (j;k) This paper is concerned with the union spΩ Tn(a) of all possible spectra that may emerge when perturbing a large n n Toeplitz band matrix × Tn(a)inthe(j; k) site by a number randomly chosen from some set Ω. The main results give descriptive bounds and, in several interesting situations, even (j;k) provide complete identifications of the limit of sp Tn(a)asn .Also Ω !1 discussed are the cases of small and large sets Ω as well as the \discontinuity of (j;k) the infinite volume case", which means that in general spΩ Tn(a)doesnot converge to something close to sp(j;k) T (a)asn ,whereT (a)isthecor- Ω !1 responding infinite Toeplitz matrix. Illustrations are provided for tridiagonal Toeplitz matrices, a notable special case. 1. Introduction and main results For a complex-valued continuous function a on the complex unit circle T,the infinite Toeplitz matrix T (a) and the finite Toeplitz matrices Tn(a) are defined by n T (a)=(aj k)j;k1 =1 and Tn(a)=(aj k)j;k=1; − − where a` is the `th Fourier coefficient of a, 2π 1 iθ i`θ a` = a(e )e− dθ; ` Z: 2π 2 Z0 Here, we restrict our attention to the case where a is a trigonometric polynomial, a , implying that at most a finite number of the Fourier coefficients are nonzero; equivalently,2P T (a) is a banded matrix. The matrix T (a) induces a bounded operator 2 n 2 on ` (N), and we think of Tn(a) as a bounded operator on C with the ` norm. Let A stand for T (a)orTn(a). The spectrum sp A is defined as usual, that is, as the set of all λ C for which A λI is not invertible. Given a complex number !, 2 − we denote by !ejek∗ the matrix that is zero everywhere except in the (j; k)entry, which is !. For a subset Ω of C, we put (j;k) spΩ A = sp(A + !ejek∗): ! Ω [2 Received by the editor August 3, 2001. 2000 Mathematics Subject Classification. Primary 47B35, 65F15; Secondary 15A18, 47B80, 82B44. Key words and phrases. Toeplitz operator, pseudospectrum, random perturbation. The work of the second author was supported by UK Engineering and Physical Sciences Re- search Council Grant GR/M12414. c 2003 American Mathematical Society 1329 License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 1330 A. BOTTCHER,¨ M. EMBREE, AND V. I. SOKOLOV (j;k) Thus, spΩ A is the union of all possible spectra that may emerge as the result of a perturbation of A in the (j; k) site by a number randomly chosen in Ω. The (j;k) purpose of this paper is to study spΩ Tn(a)forfixedj, k,andΩasn . Gohberg identified the spectrum of T (a) in 1952 [12], proving that !1 (1.1) sp T (a)=a(T) λ C a(T) : wind(a; λ) =0 ; [f 2 n 6 g where wind(a; λ) is the winding number of a (on the counterclockwise oriented unit circle) about λ.EverypointinspT (a) a(T) is an eigenvalue of finite multiplicity of T (a) or the transpose of T (a); a pointn on a(T) may be an eigenvalue or not. The spectra of the large finite Toeplitz matrices Tn(a) were studied in 1960 by Schmidt and Spitzer [26], who observed that the sets sp Tn(a)convergeinthe Hausdorff metric to a limiting set Λ(a), (1.2) lim sp Tn(a)=Λ(a): n !1 The set Λ(a) is either a singleton or the union of at most finitely many analytic arcs, each pair of which has at most endpoints in common; it can be described as k follows. We can write a(t)= k akt (t T), the sum being finite. For % (0; ), k k 2 2 1 define a% by a%(t)= ak% t (t T). Then 2P kP 2 (1.3)P Λ(a)= sp T (a%): %>0 \ From (1.1), (1.2), and (1.3) we see that in general sp Tn(a) does not converge to sp T (a). Surprisingly, pseudospectra of Toeplitz matrices behave differently. For ">0, the "-pseudospectrum sp" A is defined by sp" A = sp(A + K); K " k [k≤ with the union taken over all matrices K (of the same size as A) which induce an operator of norm at most " (see, e.g., [7], [27], [28]). Landau [20] and Reichel and Trefethen [24] (also see [2] and [7]) showed that for each ">0, (1.4) lim sp" Tn(a)=sp" T (a): n !1 It is the abyss between (1.2) and (1.4) that makes the question of the limit of (j;k) spΩ Tn(a) intriguing. To state precise results, we need some more preliminaries. For a sequence Mn 1 of nonempty sets Mn C, consider the limiting sets f gn=1 ⊂ lim inf Mn := λ C : λ is the limit of some n f 2 !1 sequence λn 1 with λn Mn ; f gn=1 2 g lim sup Mn := λ C : λ is a partial limit of some n f 2 !1 sequence λn 1 with λn Mn : f gn=1 2 g Notice that if all the sets involved are compact, then the two equalities lim inf Mn = lim sup Mn = M n n !1 !1 are equivalent to saying that Mn converges to M in the Hausdorff metric (see [14, Sections 3.1.1 and 3.1.2] or [16, Section 28]), in which case we simply write lim Mn = M; n !1 as in (1.2) and (1.4). License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use LARGE TOEPLITZ BAND MATRICES WITH A RANDOMLY PERTURBED ENTRY 1331 If λ C Λ(a), then, by (1.3), there is a %>0 such that T (a% λ) is invertible. We show2 inn Lemma 3.1 that − 1 j k (1.5) T − (a% λ)= % − djk(λ) 1 − j;k=1 with analytic functions djk : C Λ(a) C that do not depend on %.(Hereandin 1 n ! 1 what follows, we write T − ( )for(T ( ))− .) For a set M C, we define 1=M by · · ⊂ − 1=M = γ C :1+µγ =0 forsome µ M : − f 2 2 g It is well known and easily seen that if 0 Ω, then 2 (j;k) 1 (1.6) sp A =spA λ/sp A :[(A λI)− ]kj 1=Ω ; Ω [f 2 − ∈− g 1 1 where [(A λI)− ]kj is the (k; j) entry of the resolvent (A λI)− .Consequently, letting − − jk (1.7) H (a)= λ C Λ(a):dkj(λ) 1=Ω ; Ω f 2 n ∈− g we obtain (1.8) sp(j;k) T (a)=spT (a) Hjk(a): Ω [ Ω (Note that for λ outside sp T (a)wecantake(1.5)with% =1.)Formula(1.8) disposes of the \infinite volume case". In the \finite volume case" we have the following theorem, which is the main result of this paper. Theorem 1.1. Let a and let Ω be a compact subset of C that contains the 2P origin. If dkj : C Λ(a) C is identically zero or nowhere locally constant or assumes a constantn value c!that does not belong to 1=Ω,then − (j;k) jk (1.9) lim sp Tn(a)=Λ(a) H (a): n Ω Ω !1 [ We conjecture that for every a and every (j; k) the function dkj is either identically zero or nowhere locally constant,2P and hence that (1.9) is always true, but we have not been able to prove this. The results of [6] imply at least the following. Theorem 1.2. Let a be of the form 2Pq k (1.10) a(t)= akt ;p 0;q 0;apaq =0; ≥ ≥ − 6 k= p X− and let G be a bounded component of C sp T (a). The function d11 is always n nowhere locally constant in G.Ifk 1 or j 1, then the function dkj is either identically zero in G or nowhere locally≥ constant≥ in G, provided one of the following conditions is satisfied: (a) C Λ(a) is connected; (b) p orn q equals 1; (c) p + q is a prime number and p or q equals 2; (d) p + q =4or p + q =7. We remark that C Λ(a) is in particular connected if T (a) is tridiagonal (which means that p + q = 2)n or triangular (p =0orq = 0) or Hermitian. Condition (b) is equivalent to saying that T (a) is a Hessenberg matrix. Notice that we may also without loss of generality assume that p q (otherwise we may pass to the adjoint operator). The case p + q =3iscoveredby(a)for≤ p =0andby(b)forp =1, while the case p + q = 5 is contained in (a) for p =0,in(b)forp =1,andin(c) for p = 2. We have no result in the case p + q = 6 (unless p =0orp =1). License or copyright restrictions may apply to redistribution; see https://www.ams.org/journal-terms-of-use 1332 A. BOTTCHER,¨ M. EMBREE, AND V. I. SOKOLOV Since sp T (a) is in general much larger than Λ(a), we see from (1.8) and (1.9) that (j;k) spΩ generically behaves discontinuously when passing from large finite Toeplitz matrices to infinite Toeplitz matrices.