Unified Patterns for Realtime Interactive Simulation in Games and Digital Storytelling

Dieter Schmalstieg

Graz University of Technology

Abstract—This paper discusses the state of the art in realtime interactive simulation patterns, which are essential in the development of game engines, digital storytelling, and many other graphical applications. We show how currently popular patterns have evolved from equivalent, or, at least, very similar concepts, and are deeply related. We show that game engines do not need to choose a particular pattern, but can easily combine all desired properties with little or no overhead.

Realtime interactive simulations, such as simulation should be flexible in accommodating computer games or cosimulations,1 typically use new entities and new tasks. entities and tasks as their building blocks. A Contemporary game engines, such as Uni- game world or other graphical environment is ty3D, favor the entity/component/system (ECS) populated by entities such as creatures or treas- pattern,2 which makes it easy to extend enti- ures. Many entities coexist, over longer periods ties and tasks independently. ECS decouples of time, and the digital narrative emerges as a entities and tasks, which is often seen as a consequence of their simulated interactions. major advantage over traditional object-ori- The challenge from a programming point of view ented programming techniques. However, it is scalability: We want to make it convenient for does not support communication between the programmer to express the desired simula- 3 entity groups. Mercury has recently shown tion in terms of entities and tasks, and the how Unity3D can be extended with group com- munication based on dataflow, but Mercury still requires to know communication patterns in advance. Figure 1. Relationship of the architectures (blue boxes) discussed in this article. Arrows denote the addition of a particular concept.

In this article, we compare several popular register its interest with many subjects, and a communication patterns and show how the com- subject can receive interest from many observ- bination of ECS with dynamic group communica- ers. Moreover, multiple observers can be tion unifies all previously proposed patterns arranged in sequence, allowing indirect notifica- into a coherent whole (Figure 1). tion, provided that events are automatically for- warded to the next observer in the chain. The resulting structure is a dataflow graph (or pipes- NOTIFICATION and-filters pattern),5 a directed graph with enti- In object-oriented languages, entity types are ties as nodes and subject-observer relationships typically derived from an abstract base class, as edges. while tasks are implemented as entity methods. A naive dataflow schedules nodes in depth- Direct method invocations implies that the invok- first order, as with direct method invocations. ing entity must frequently poll the invoked entity Alternatively, a more sophisticated scheduling to learn about any new information. Polling is can interpret the nodes as states in a state wasteful, if there are many entities or infrequent machine. Notification starts at a source node and changes. Moreover, the schedule in which tasks is propagated based on state machine rules (see are carried out is bound to the invocation Figure 2). For example, breadth-first scheduling sequence. We could inadvertently trigger a deep would require that all nodes with a shorter dis- sequence of method invocations, starving more tance to the source must be processed first, urgent tasks. before a given node may be processed. More com- Consequently, we replace polling with noti- plex rules can impose arbitrary topological order- fication: Entities communicate by passing mes- ing on the notifications. For example, one may sages about events that happened in the require that a node has events pending on some simulated world. In order to receive events, an or all of its incoming connections. Such a flexible observer entity registers itself with a subject scheduling can be implemented using a visitor pat- 4 entity to receive notifications. Notification sepa- tern,4 an iterator which traverses a graph accord- rates event creation and consumption. Notifica- ing to a rule set and delivers notifications by tions can be delivered in a different order than invoking a virtual function on each visited node. the one in which the corresponding events Dataflow has been used in many interactive were created. This freedom of reordering notifi- applications. Visualization or image processing cations enables the development of a large vari- pipelines can be set up as dataflow,6 represe- ety of scheduling strategies, suiting the needs of nting filters, transformations, and encodings as diverse flavors of simulation. nodes. Input processing in virtual and augm- ented reality is often specified as dataflow,7 to DATAFLOW stream input from multiple sensors through fil- The observer pattern makes it easy to control ters, fuse operations, and apply geometric notification propagation. A single observer can transformations. Figure 2. (Top) Augmented reality in a folklore museum. Left: The visitor retrieves the flat iron for finding out the initials of the bridal pair; Middle: The visitor heats the iron in the tailor’s oven. Right: The visitor learns about the “wedding rider” custom. (Bottom) Finite state machine expressing the application logic of the folklore museum experience.

A scene graph,8 a common data structure for a publish/subscribe architecture:5 Entities pub- visualization and games, can be interpreted as a lish to a notification channel, while other entities special case of dataflow (see Figure 3). The subscribe to this channel. The subscription can visual representation of a scene naturally lends be determined by explicitly associating entities itself to a hierarchical representation, with with channels, but a more powerful scheme nodes representing scene objects (triangles, tex- determines subscription based on dynamic fil- tures, shaders, etc.) or geometric transforma- tering of attributes, such as proximity. An inter- tions. A visitor traverses the nodes of the scene mediary object, the broker, collects events and graph, accumulates a transformaton matrix and delivers notifications. The broker can implement calls a rendering method on each node. arbitrary scheduling strategies and even provide persistent storage. PUBLISH/SUBSCRIBE When the broker collects and buffers many Dataflow supports indirect notification, but events from many entities, temporal ordering still requires explicitly setting up the links of events becomes essential. Proper temporal between entities to send notifications between ordering requires that notifications for events entities. Setting up links for static relationships is are delivered in the half-order induced on the easy, but, if entities act autonomously, associa- events’ time-stamps. tions change in dynamic, unpredictable patterns. In such cases, we would like to specify enti- MODEL/VIEW/CONTROLLER ties to be notified by filtering attributes or speci- If event processing and rendering are cou- fying interests, rather than by enumerating pled together in the same loop, either one of entities directly. Instead of a visitor, we need the two tasks can become a bottleneck. Figure 3. Flow visualization around a space shuttle (left) is modeled as a hybrid of scene graph (middle) and a dataflow graph (right).

Simulations should, therefore, incorporate a node to node during visitation, a conventional decoupled simulation pattern9 that assigns sep- visitor cannot be used. Such a case requires arate threads of control to concurrent tasks. double dispatch,10 the ability to vary both entity In a decoupled simulation, a broker may place and task dependent on the use case. There are its event queues in shared memory, accepting several ways how double dispatch can be events from entities belonging to one thread implemented: and notifying entities belonging to another  thread. Provided we use a strictly asynchro- Multiple inheritance can support double dis- nous scheduling, where no return notifications patch to some degree, but can lead to compe- are required, each task may iterate over the tition among inherited implementations received notifications at its own pace, without (“deadly diamond of death”).  blocking other tasks. Reflection can extract a method signature In a simple architecture, a main thread may from events and invoke a corresponding be responsible for brokering events, while other, method. Reflection can either be natively sup- secondary threads are spawned on demand for ported by the language (e.g., C#) or emulated compute-intensive activities and retire after they by parsing a textual event representation. have fulfilled their goal. In a more complex archi- Using reflection is generally considered too tecture, multiple threads may permanently tend costly for processing large amounts of events.  to activities such as rendering, AI, animation, A signal/slot pattern can be used, which physics, and so on. allows registering methods of the notified The model/view/controller (MVC) pattern5 object to a particular signal (event). If the describes a common form of decoupled simula- programming language does not support a tion. The model is a common store for the enti- signal/slot construct, it can be implemented ties, while controller and view are tasks by using templates or building a table of func- observing of the model. The view is responsible tion pointers.  for rendering, while the controller is responsible The VIGO (views/instruments/governors/obj- 11 for processing notifications. If needed, MVC can ects) pattern extends MVC with a form of also use multiple views and multiple controllers double dispatch where behaviors are split per model. across two independently varied types of controllers, called “governors” and “instru- ments.” Instruments represent tasks, while ENTITY/COMPONENT/SYSTEM governors are mediators4 facilitating appro- While MVC can easily provide decoupled sim- priate reactive behaviors of entities. ulation of multiple tasks, a conventional visitor  Inheritance can be replaced by aggregation. is designed for one specific task. The visitor 12 needs to know which virtual method to call on The entity/component pattern takes the lat- the visited entities. If the set of tasks is not ter approach, by aggregating components inside known in advance, or the task changes from an entity. Each component is responsible for a Figure 4. ECS pattern (bottom) can be explained as a combination of concepts from the model/view/ controller (left) and the entity/component (right) pattern. In ECS, a given set of entities and tasks (top) is represented inside per-task systems. specific task. Entities and components are task are represented, leading to enhanced data derived from an entity base class and a compo- locality if a task requires many entities to nent base class, respectively. Hence, implemen- collaborate. tations of entity and component can be varied An alternative implementation of ECS can independently to achieve double dispatch. let systems handle events directly, rather than When an entity receives an event, the entity delegating the event to a component.13 In this searches its component directory for a suitable variant of ECS, the system more closely resem- component and passes the event. bles the controller of MVC, by using compo- The entity/component pattern provides an nents as mere data containers. The system elegant solution for double dispatch, but it is not cannot benefit from double dispatch afforded ideal for data locality, since it organizes the data by a class hierarchy of components. However, by an entity and not by task. Encapsulation of using containers as pure data stores has dis- components inside entities implies that poten- tinct advantages. First, it becomes easier to tially very different data elements are aggregated use relational databases for persistent storage. inside one entity. A better data organization Second, a task that needs to run intensive would result from following data-oriented pro- processing on a large number of identically gramming principles and organizing data by structured components (e.g., finite element task, not by an entity. This principle is known simulation) is more efficiently carried out on from column-oriented stores or the structure-of- the system level without invoking virtual meth- arrays pattern. ods. Third, a system that just wraps around an We can combine the entity/component existing library is essentially implementing a pattern with aspects of the MVC pattern to mediator pattern4 and has no use for obtain the ECS pattern.2 Entities become noth- components. ing more than a unique entity id. Tasks are split into a model part (the component) and a BROKERED ECS controller part (the system). A rendering sys- From the architectures discussed so far tem corresponds to the view of MVC. (see Figure 4), we arrive at three distinct pat- Entities communicate by double-dispatch, as terns: (1) ECS is the preferred implementation in entity/component, but with inverted order: pattern for modern game engines. (2) Publish/ first, on the system (task) level and, second, on subscribe is a pattern often employed in net- the component (entity) level. Like in MVC, every worked systems, such as the internet of things system can run in its own independent thread, or multiuser virtual reality. (3) Dataflow is facilitating decoupled simulation. Inside a sys- commonly used in visualization and interac- tem, only components belonging to the same tion. Mercury3 combines ECS with dataflow, Figure 5. Dataflow architecture can be integrated into a publish/subscribe architecture by extending the broker. In the Brokered ECS variant, direct notification is replaced with the extended broker.

but lacks a more powerful publish/subscribe all discussed approaches: It supports all forms mechanism. This observation prompts the of notification, topological and temporal order- questions how difficult it would be to combine ing, decoupled simulation, double dispatch, all three patterns. and data-oriented programming. Since a bro- Let us consider a combination of dataflow kered ECS is only marginally more complex to and publish/subscribe (see Figure 5, left). Obvi- build than the previous approaches, we con- ously, notification can be sent to direct and indi- clude that making a hard choice between old rect neighbors of an entity through a channel. and new programming style in realtime interac- The channel access may require an additional tive simulation is not necessary. lookup compared to a callback mechanism, but if we assume at least one such lookup is required REFERENCES for using a virtual method, the overhead of going through a channel can be hidden with 1. C. Gomes, C. Thule, D. Broman, P. G. Larsen, and proper programming techniques. Moreover, an H. Vangheluwe, “Co-simulation: State of the art,” extended broker can enforce topological order- CoRR, abs/1702.00686, 2017. ing by checking the constraints imposed on a 2. S. Bilas, “A data-driven game object system,” Blog node before an event is delivered. Hence, we can Post, 2002. Available at: http://gamedevs.org/uploads/ easily integrate dataflow, state machines and data-driven-gameobject- system.pdf publish/subscribe. 3. C. Elvezio, M. Sukan, and S. Feiner, “Mercury: A Inserting such an extended broker into an ECS messaging framework for modular ui components,” in completes our system integration (see Figure 5, Proc. CHI Conf. Hum. Factors Comput. Syst., 2018, right). A brokered ECS routes notifications pp. 588:1–588:12. between components inside separate systems 4. E. Gamma, R. Helm, R. Johnson, and J. Vlissides, through a channel. Membership in channels facil- Design Patterns: Elements of Reusable Object- itates arbitrary group notification. All forms of Oriented Software. Boston, MA, USA: Addison- notification benefit from the double dispatch Wesley, 1995. capability inherited from Entity/Component. 5. F. Buschmann, R. Meunier, H. Rohnert, P. Sommerlad, and M. Stal, Pattern-Oriented Software Architecture - Volume 1: A System of Patterns. Hoboken, NJ, USA: CONCLUSION Wiley, 1996. This paper establishes a unified view on 6. W. Schroeder, K. M. Martin, and W. E. Lorensen. The several popular architectural patterns for Visualization Toolkit (4th Ed.): An Object-oriented game engines and other graphical simulations. Approach to 3D Graphics, 2006. We show that these patterns are not compet- 7. G. Reitmayr and D. Schmalstieg, “Opentracker - A ing, but deeply related. Consequently, we flexible software design for three-dimensional arrive at an integrated approach, brokered interaction,” Virtual Reality, vol. 9, no. 1, pp. 79–92, ECS, which combines the desired properties of Dec. 2005. 8. P. S. Strauss and R. Carey, “An object-oriented 3d graphics toolkit,” SIGGRAPH Comput. Graph., vol. 26, no. 2, pp. 341–349, Jul. 1992. 9. C. Shaw, M. Green, J. Liang, and Y. Sun, “Decoupled simulation in virtual reality with the mr toolkit,” ACM Trans. Inf. Syst., vol. 11, no. 3, pp. 287–317, Jul. 1993. 10. D. H. H. Ingalls, “A simple technique for handling multiple polymorphism,” in Proc. Conf. Proc. Object- Oriented Program. Syst., Languages Appl., 1986, pp. 347–349. 11. Michel Beaudouin-Lafon Clemens Klokmose. “Vigo: Instrumental interaction in multi-surface environments,” in Proc. ACM CHI, 2009, pp. 869–878. 12. C. Stoy, “Game object component system,” in Game Programming Gems 6, 2006. 13. A. Martin, “Entity systems are the future of mmog development,” Blog Post, 2007. [Online]. Available: at: http://tmachine.org/index.php/2007/09/03/entity- systemsare- the-future-of-mmog- development-part-1/