Rendering Pipeline Modelled by Category Theory Jiríˇ Havel Adam Herout Faculty of Information Technology Faculty of Information Technology Brno University of Technology Brno University of Technology ihavel@fit.vutbr.cz herout@fit.vutbr.cz ABSTRACT This paper describes basic concepts from category theory, which are commonly used in functional programming. These con- cepts are applied to shader programming and to the rendering pipeline and the whole rendering pipeline is formally modelled using category theory. This model can be used for more abstract and formal approach to shader programming. Mathemat- ical formalization of the rendering pipeline and its stages can be helpful in shader compiler design, for proving algorithms, complexity analysis, and other tasks. Keywords: Rendering, Shaders, Category Theory 1 INTRODUCTION 2 CATEGORY WITH STREAMS Category theory [9, 3] is an abstraction of mathemati- A category (C) consists of a set of objects (obj C) and cal structures and relations between them. It started as a a set of arrows or morphisms (arr C) that link these "generic abstract nonsense", but now it is heavily used objects. An arrow f : A → B has a domain A and a not only in mathematics, but also in computer science codomain B from obj C. The set of arrows from A to B and especially functional programming. Many categor- is denoted as HomC(A,B). Arrows must be composable ical concepts give rise to common pieces of code used (1a), every object must have one identity arrow (1b) to combine computations together. >From these ab- (1c) and arrow composition must be associative (1d). stractions, especially functors and monads are almost ubiquitous [7]. ∀ f : A → B,g : B → C ∃(g ◦ f ) : A → C (1a) Although the rendering pipeline is programmable, its ∀O ∈ obj C ∃1O : O → O (1b) overall structure is basically fixed. The design of pro- ∀ f : A → B,1B ◦ f = f ◦ 1A = f (1c) gramming languages for shader programming perfectly h ◦ (g ◦ f ) = (h ◦ g) ◦ f (1d) corresponds to the structure. Some experimental shad- ing languages tried to offer a slightly more abstract A classical example is the category Set, whose objects 1 way. Two notable examples are the Gpipe library for are sets and arrows are mappings between those sets. Haskell and Sh [5, 4] for C++. Both are, however, lim- N-ary functions and tuples are modelled using prod- ited to OpenGL 2 or DirectX 8 functionality. ucts – composite objects. Product P is an object con- The Sh library views shaders as objects, that can be sisting of objects Oi,i = 0,...N with projection arrows combined using two operations – serial and parallel (pi : P → Oi) that extract its member objects. For ev- composition. These two operations, however, perfectly ery object Z that has arrows ai : Z → Oi to all members correspond to a category with products, which will be of P, exactly one arrow aP to the product exists, such described in the next section. that ai = pi ◦aP, see Figure 1. An exaple of the product The GPipe library achives the same functionality by type can be the structure data type, that is a product of a different approach. It operates on streams of primi- its elements. When describing function, the arrow aP is tives and transforms them by series of functions – using a combination of functions returning the elements of a functors from the category theory. structure to a function returing the whole structure. The rest of the paper will discuss correspondences between those approaches. Section 2 will introduce a category-based model of the rendering pipeline stages and extend programmingof pipeline stages to a descrip- tion of the whole pipeline. Section 3 will raise the ab- straction of the pipeline description to a composition of stream transformers. Section 4 summarizes the intro- duced categories as a model of the rendering pipeline. Figure 1: Product in a category Let us define a category of GPU programs called 1 http://www.haskell.org/haskellwiki/GPipe Gpu. Objects of Gpu are basic and structured datatypes of graphical shaders and arrows are func- B. Arrows of this category are called natural transfor- tions (both built-in and composite). Both structured mations. For every functor F and G from BA, natu- datatypes (structure and array) fullfil the requirement ral transformation φ : F → G and arrow f ∈ A, must for product types2 as do the basic vector types. With φ ◦ F( f )= G( f ) ◦ φ. streams from the following subsection, every possible Natural transformations change only the structure of shader will be an arrow in Gpu. an abstract data type, but the elements of the type are left unchanged. For example, a transformation between 2.1 Streams are Functors an array of streams and a stream of arrays is a natu- Functor is a structure-preserving mapping be- ral transformation. Natural transformations are heavily tween categories. Functor F : A → B be- used in the following subsections. tween categories A and B consists of a map- 2.2 Streams are Monads ping Fobj : obj A → obj B and mappingss → Monad is a special type of functor used in functional FA1,A2 : HomA(A1,A2) HomB(Fobj(A1),Fobj(A2)) for every pair of objects A1, A2 from A. Functors must programming to represent computations and control also preserve arrow composition and identity arrows. structures, to embed side effects, or model a process- Homogenous abstract data types like lists, stacks, or ing pipeline. queues are constructed by functors. The Fobj creates Monads in category theory is a functor F, together structured types from the basic ones. The mappings for with two natural transformations η :1C → F and µ : arrows convert simple arrows to arrows working with F2 → F (F2 = F ◦ F). The corresponding triplet in new types – usually doing the same for every element functional programming consists of a functor F and of the new type. In functional programming, this fam- mappings unit : A → F(A) and join : F(F(A)) → F(A) ily of mappings is denoted by map. map : (X → Y) → [6]. (F(X) → F(Y)), where F is a functor, i.e., for a given Mapping unit creates the monad type from arrow on the simple data type, map defines the corre- one element and join merges two layers of sponding arrow on the structured data type. the monad to one. In the category Gpu, the In shader programming, one abstract data type is transformation unit : X → Stream(X) creates a ubiquitous – the stream of values (vertices, primitives, stream with a single element. The transformation fragments). It is a sequence of elements, that is, join : Stream(Stream(X)) → Stream(X) joins a stream however, never handled directly, but implicitly by the of streams to one single stream by concatenation. streaming nature of the rendering pipeline. Stream In functional programming, the bind transformation types in the category Gpu are constructed using a is used more than join and it better describes the prop- functor Stream : Gpu → Gpu. This functor creates erties of monads. streams of elements of any basic type (and recursively bind : (X → F(Y)) → (F(X) → F(Y )) streams of streams). For every X and Y from obj Gpu, the mapping for arrows is simply defined as bind( f )= join ◦ map( f ) → map : (X → Y) → (Stream(X) → Stream(Y)) In Gpu, the transformation bind : (X Stream(Y)) → (Stream(X) → Stream(Y)) uses map( f )(x1,...,xn) = ( f (x1),..., f (xn)). the provided mapping to transform a stream. The type shows that every element of the input stream is The mapping map( f ) transforms every stream element used to create a new stream and such new streams are by the function f , see Figure 2. concatenated together. In other words, 0 − N elements can be created from every single input element and the input elements are processed separately, as shown by Figure 3. Figure 2: Functorial transformation principle. Streams are not the only functors in the category Gpu, but arrays and basic vector types are functors as well. For these data types, map works similarly to its Figure 3: Monadic transformation schema. stream counterpart. A perfect example of monadic processing of a stream For categories A and B a category BA exists: the is the geometry shader. From every primitive of the functor category, whose objects are functors from A to input stream, zero or more primitives are generated and those new streams are concatenated. Also, the fragment 2 Objects like sums (discriminated unions) or exponentials (partially applied functions) are hard to express on GPU, so will not be used in shader can be described by a monad, as it can output this paper. empty streams or streams with a single element. 2.3 Streams are Comonads For a category C with monad M, which contains ar- rows of type A → M(B),A,B ∈ obj C, exists another Functor and monad are sufficient to describe stream category K, transformers that access single elements of a stream. For accessing multiple elements, a categorical dual to obj K = obj C a monad can be used – the comonad. Comonads can HomK(A,B)= HomC(A,M(B)) represent some information in a context. In the case of streams, the context of each element are the neighbor- The arrow composition in K is defined as gK ◦ fK = ing elements. bind(gC) ◦ fC, so kategory K expresses composition of As dual functor to a monad, comonad is a func- stream transformations. This category is called Kleisli tor with two natural transformations in opposite di- category of C. Dual to Kleisli category, also the CoKleisli category rection as the monad. These are extract : F → 1C and duplicate : F → F2.