A practical framework for infinite-dimensional linear algebra Sheehan Olver∗ Alex Townsendy ABSTRACT as discretize-then-solve. That is, the underlying infinite- We describe a framework for solving a broad class of infinite- dimensional operator is first approximated by a finite-dimensional dimensional linear equations, consisting of almost banded matrix, before the resulting linear system is solved by a stan- operators, which can be used to resepresent linear ordinary dard linear algebra method that is either direct, such as differential equations with general boundary conditions. The Gaussian elimination, or iterative, such as conjugate gradi- framework contains a data structure on which row opera- ent. tions can be performed, allowing for the solution of linear In contrast, we advocate an entirely different approach equations by the adaptive QR approach. The algorithm that solves the equation as an infinite-dimensional problem achieves Onopt complexity, where nopt is the number of and never discretizes the operator itself. To accomplish this degrees of freedom required to achieve a desired accuracy, task, we represent the (infinite-dimensional) operator by a which is determined adaptively. In addition, special tensor suitable data structure that supports row manipulations di- product equations, such as partial differential equations on rectly on the representation of the operator, using lazy eval- rectangles, can be solved by truncating the operator in the uation to automatically extend the data in the representa- tion as needed. The row operations can be used to partially y-direction with ny degrees of freedom and using a general- ized Schur decomposition to upper triangularize, before ap- upper-triangularize the operator, and, for a large class of plying the adaptive QR approach to the x-direction, requir- problems (in particular non-singular ODEs), we can at some ing On2 nopt operations. The framework is implemented in point perturb the right-hand side by a small amount so that y x the still infinite-dimensional problem can be solved exactly the ApproxFun package written in the Julia programming language, which achieves highly competitive computational via a (finite-dimensional) back substitution step. The mathematical ground work for this approach is (F. costs by exploiting unique features of Julia. W. J.) Olver's algorithm [11], which considers the solution of inhomogeneous three-term recurrence relationship. This Keywords is equivalent to solving an infinite-dimensional linear system Chebyshev, ultraspherical, partial differential equation, spec- involving a rank-1 perturbation of a tridiagonal operator: tral method, Julia 0 1 0 1 0 1 b0 b1 b2 b3 ··· u0 f0 Bγ1 α1 β1 C Bu1C Bf1C 1. INTRODUCTION B C B C B C B γ2 α2 β2 C Bu C Bf C Linear equations play a fundamental role in scientific com- B C B 2C = B 2C : B :: C B C B C puting, with the classical examples including the numerical B γ3 α3 :C Bu3C Bf3C @ A @ A @ A solution of boundary value ordinary differential equations, :: :: : : elliptic partial differential equations and singular integral : : : : equations. Practically all numerical methods for solving lin- The key observation is that the infinite-dimensional linear ear differential equations | e.g., finite difference, finite ele- system can be solved by Gaussian elimination without piv- arXiv:1409.5529v1 [math.NA] 19 Sep 2014 ment, collocation and Galerkin methods | can be described oting and that convergence to the minimal solution of the ∗School of Mathematics and Statistics, The University of system | roughly, the solution (provided it exists) with the Sydney, Sydney, Australia. ([email protected]) fastest decaying entries | can be inferred as part of the y Department of Mathematics, Massachusetts Institute of algorithm. Back substitution then proceeds by perturbing Technology, 77 Massachusetts Avenue Cambridge, MA the right-hand side, as opposed to changing the infinite- 02139-4307. ([email protected]) dimensional operator. In the functional analysis setting, where the operator is assumed to be invertible between two spaces, the minimal solution is the unique solution to the lin- ear equation. This approach was extended by Lozier to more Permission to make digital or hard copies of all or part of this work for general banded operators [9]. However, Gaussian elimina- personal or classroom use is granted without fee provided that copies are tion without pivoting is prone to numerical instability, and not made or distributed for profit or commercial advantage and that copies with this in mind the authors derived an adaptive QR ap- bear this notice and the full citation on the first page. To copy otherwise, to proach [13], using Givens rotations for the solution of linear republish, to post on servers or to redistribute to lists, requires prior specific ordinary differential equations (ODEs). The complexity of permission and/or a fee. opt Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$15.00. Olver's algorithm and the adaptive QR approach is O n , where nopt is the number of coefficients calculated, as deter- which returns a vector of the entries in the columns specified mined automatically by the convergence criteria. (Through- by cr. Similarly, each subtype of BandedOperator overrides out, an integer with a superscript \opt" is a number that is a routine called addentries!, which adds entries to specified determined adaptively as dictated by the particular prob- rows of a (finite) banded array1, and overrides a routine lem.) called bandinds that returns the band range of the operator Similar in spirit to the current work is Hansen [7], which represented as a tuple (a; b). investigated the infinite-dimensional QR algorithm for spec- As an example, consider representing functionals and op- tral problems, though focusing on theoretical rather than erators that act on vectors of Taylor series coefficients, i.e., > practical matters. Our operator algebraic framework is heav- vectors of the form (u0; u1; u2;:::) that correspond to the P1 k ily influenced by the chebop system [4], which is part of series k=0 ukz . Evaluation at a point z is the functional 2 Chebfun [5], and provides an infinite-dimensional feel to Bz , [1; z; z ;::: ] and thus we can create a subtype Tay- the user, though the underlying collocation method is the lorEvaluation, with single field z, that implements traditional approach of discretize-then-solve. Finally, the second author and Trefethen investigated continuous ana- getindex(B::TaylorEvaluation,cr)=B.z.^(cr-1): logues of matrix algorithms [16], where the emphasis is on We can also implement a TaylorDerivative operator to rep- representing smooth bivariate functions rather than opera- resent the banded operator, defined by e>De = k and tors. k k+1 zero otherwise, with band range 0 : 1. Finally, we can rep- In this work, we exploit the applicability of the adaptive resent multiplication by a polynomial of finite degree, says QR method for a general class of linear operators: banded Pm−1 k a(z) = k=0 akz with the Toeplitz operator defined by operators except for possibly a finite number of dense rows. > However, to make this competitive and useful for general ek T [a]ej = ak−j for 0 ≤ k − j ≤ m − 1, and zero otherwise. problems requires the following components: This is encoded in a TaylorMultiplication operator, which has a single field containing the coefficients of a as a vector 1. Abstract data types that can be overriden to represent of length m and a band range (1 − m):0. arbitrary (typically unbounded) banded operators and While our operators always act on infinite-dimensional dense functionals, as well as data structures to allow vectors, the entries of those vectors can represent coeffi- their algebraic manipulation. cients in many different bases. For example, in [13] vectors 2. Fast linear algebra on infinite-dimensional operators. represent expansion coefficients in the Chebyshev or ultras- This requires a carefully managed data structure that pherical basis. To ensure that the domain and range of two can encapsulate the full infinite-dimensional operator operators are consistent when, for instance, adding them to- at each stage of the linear algebra routines. gether, each operator must know the basis of its domain and range. This also allows us to automatically convert between Unfortunately, the specialized data structures that we de- bases to ensure that any operation can be performed in a velop also require very specific implementations of linear consistent manner. Therefore, we have a FunctionSpace algebra routines, prohibiting the traditional approach of re- abstract type so that operators can override domainspace ducing the problem to finite-dimensional linear algebra solv- and rangespace routines that return specific domain and able by LAPack. In [13], the C++ language was used to range spaces. When it exists, a banded conversion operator partially implement the framework for some simple exam- is implemented to convert between two spaces. ples; however, adding new operators required a complete re- The final components are structures that allow functional compilation, which is prohibitively time consuming for prac- and operator algebra. This consists of a PlusOperator, which tical use. As an alternative, the Julia programming lan- contains a list of BandedOperators that have the same range guage [2] provides a natural environment for implementing and domain spaces. The command \+" is then overridden both data structures and linear algebra algorithms. Fur- for BandedOperators, with an additional step of promoting thermore, the support for multiple dispatch allows for the the domain and range space whenever a banded conversion easy construction of data structures, and linear algebra can operator is available. Similarly, a TimesOperator is con- be performed remarkably efficiently due to on-the-fly com- structed to represent multiplication of operators, and \∗" is pilation. The ApproxFun package [12] implements the pro- similarly overloaded to promote spaces to ensure compati- posed framework in Julia. bility. Note that, if A and B have band range a : b and c : d, respectively, then the band range of A*B is (a + c):(b + d).