Visualising air pollution datasets with real-time game engines

Uli Meyer1, Jonathan Becker1, and Jessica Broscheit1

CSTI, Hamburg, Germany, [email protected], WWW home page: https://csti.haw-hamburg.de/ Hamburg University of Applied Sciences, Berliner Tor 5, 20099 Hamburg, Germany

Abstract. Visualising Volunteered Geographic Information (VGI), in- cluding air pollution data, as an explorative tool in the context of work- shops and maker labs, requires a technology that has a low entry-level, but provides a powerful interactive prototyping framework. We describe the potential of real-time computer game engines as visualisation tools for interdisciplinary cooperation between non-experts and experts. We discuss how properties of air pollution, including invisibility, pervasive- ness and its ability to permeate organisms, can be visualised with particle systems, and outline two use cases for different output devices, including AR and VR. ...

Keywords: visualisation, air pollution, real-time engine, game engine, particle system, augmented reality, virtual reality

1 Introduction

Visualising Volunteered Geographic Information (VGI), including air pollution data, as an explorative tool in the context of workshops and maker labs, re- quires a technology that has a low entry-level, but provides a powerful inter- active prototyping framework. Ideally, this framework allows non-experts and experts to work together on interdisciplinary projects that support experiential learning [5]. In the context of the serious game movement and for interactive pro- totyping, real-time computer game engines have proven to be such a versatile tool [13][16][20]. While they cannot compute scientifically accurate simulations, during development processes they provide immediate and continuous real-time visualisations within the 3D view port. Visualising air pollution data sets can have diverse functions, ranging from declarative to experimental [1]: Simplified visualisations can serve as a warning, for example in big cities, or for sensitive persons [15][17]. Scientific visualisations give insight into behaviours and compositions of air pollution over time. Visu- alisations can also raise awareness for air pollution in more pedagogic contexts [10], or they can be used as a tool for community activism and policy change [8].

In: Rocha Á., Adeli H., Reis L., Costanzo S. (eds) New Knowledge in Information Systems and Technologies. WorldCIST’19 2019. Advances in Intelligent Systems and Computing, vol 932. Springer, 2019 2 Uli Meyer et al.

Depending on the function, different types of visualisation, from highly abstract 2D diagrams to interactive 3D info graphics and immersive VR experiences, are available [4][6][11]. This paper describes the technological potential of game engines for experimen- tal visualisations of complex air pollution datasets, and proposes two possible use cases for such a visualisation. Part 2 introduces visualisation terminology and discusses the specifics of air pol- lution visualisation. Part 3 gives an introduction to game engine technology for visualisation purposes and Part 4 outlines two potential use cases for experimen- tal air pollution visualisation in a game engine.

2 Data Visualisation

Data visualisation implies a transfer of abstract information, usually via a nu- merical transformation, into a more concrete form such as geometric shapes [Fig.1].

Fig. 1. Visualisation process [9].

2.1 Terminology

Depending on the function, Scott Berinato [1] classifies visualisations along two axes: from “conceptual” to “data-driven” and from “declarative” to “ex- ploratory”. According to his classification system, data-driven visualisations can be described as more “declarative” or as more “exploratory” [Fig. 2]. Traditionally, data visualisation meant static 2D diagrams [6]. With the advent of 3D graphics since the mid-20th century, elements such as movement, real-time rendering and interaction were introduced [4]. The development from 2D graph- ics to 3D graphics brought new possibilities, but also new problems: While 2D graphics are easier to read and more condensed, their “cleanness” and simplicity Visualising air pollution datasets with game engines 3 might be misleading, especially for organic data. 3D visualisations, on the other hand, can be harder to read, for example by creating visual overlap. They need very precise planning, or additional movement in the form of pre-rendered ani- mation. Since the late 1990s, real-time movement and interaction became available with the introduction of real-time engines. Real-time visualisations expose one or several parameters that can be influenced by the user in some way, for example camera position, object translation, rotation, scale, and so on. Data visualisation in virtual reality (VR) constitutes an extreme case of real-time interactivity, as the user can move through, or even climbs on diagrams in virtual space.

Fig. 2. Visualisation categories [1].

2.2 Visualising air pollution data If we follow Berinato [1], visualisations of air pollution datasets can be cate- gorised along an axis from declarative to exploratory. Declarative visualisations of air pollution would be what he calls “everyday dataviz”, e.g. the 2D dia- grams one finds in the media or in textbooks. These declarative visualisations are also used for warning systems in cases where the existence of air pollution is undisputed, and where concrete measures need to be taken, often within a very short time frame [Fig. 3]. While declarative visualisations are usually rendered as static 2D graphics, exploratory visualisations often contain interactive elements, for example sliders or real-time animations [19] [Fig. 4]. Visualisations that exist on the boundary between declarative and explorative can be found where the degree or danger of air pollution is under debate. Their purpose is usually educational or activist. They often contain hybrid combina- tions of 2D abstractions and more concrete 3D elements, such as bar charts in 3D space. Or they visualise air pollution data as clusters of “balls” that can pile up over time [Fig. 5]. 4 Uli Meyer et al.

Fig. 3. Air pollution monitoring in China [15], Air pollution warning app [17].

Fig. 4. Interactive visualisation of air pollution [3].

Fig. 5. Hybrid air pollution visualisations [3]. Visualising air pollution datasets with game engines 5

Fig. 6. WIFI visualisation[12], geospatial visualisation[7].

While these hybrid displays are often striking, and can have an immediate effect on the viewer, they somewhat obscure the properties of air pollution. By abstracting air pollution - an inhaled gaseous mixture that contains fluids and tiny particles - as solid objects, these images suggest a degree of visibility and potential control that does not in fact exist. But evoking simple solutions and a “lets-do-this” attitude might be a valid strategy for marketing purposes, which includes fund-raising or activism. Air pollution is mostly invisible and volatile, and it permeates organisms. Actual or metaphoric visualisations of the pervasive nature of air pollution are relatively rare [2]. Two augmented reality (AR) applications can serve as models for visualising the invisible, in these cases WIFI density or gravity in space. They overlay im- ages or real-time video of specific physical environments with areas or clouds of colour in space. Different colours indicate different types of quality [Fig. 6]. But even these applications maintain a separation between the user and the coloured clouds. They are “over there”, not all around the user. An example for the vi- sualisation of real-time immersion in an air/water mixture would be the Rain art installation at the Barbican Centre in London [18]. There visitors can move freely through physical water drops in space, without getting wet. The effect is similar to “diving into” particle simulations in VR experiences such as The Blue [21] or Cosmic Sugar [14] [Fig. 7]. While The Blue immerses users in a simu- lated underwater environment, filled with floating particles and sea life, Cosmic Sugar surrounds them with more abstract “gaseous” particles. Inside both VR experiences, the user can interact with the particles in real-time by using a hand controller. In the following part, we describe how game engines can be used to create ex- plorative visualisations of air pollution data.

3 Technology

3.1 Game Engines Game engines are real-time engines that provide software-development environ- ments, or frameworks, for building computer games and other real-time appli- cations. Due to the popularity of computer games, and the fact that several 6 Uli Meyer et al.

Fig. 7. Rain installation at the Barbican, London (2013)[18], The Blue (2016)[21]. game engines are free for non-commercial purposes, game engines are an ideal technology for entry-level development, prototyping and experimentation. By playing computer games, many people have already come in contact with games engines. They can provide an explorative framework for workshops, makerlabs or private houses, both as an agile and experimental prototyping environment, and for creating interactive applications and visualisations. 3D game engines that are currently free for non-commercial purposes include Unity 3D, Unreal Engine, and CryEngine. Game engines consist of a 3D view port where the user can assemble 3D objects with simple drag-and-drop interactions. Other windows and tabs contain menus for influencing position, scale and rotation, or animation and lighting. All pa- rameters can be made interactive, i.e. the user can influence them in real-time, either by typing in numbers, or by affecting them in the 3D view port with a mouse or controller. For some engines, such as Unity, basic, entry-level coding in C# or Java Script is needed, while others contain node-based, visual scripting systems (e.g. Unreal Engine’s Blueprint system) that are easy to learn. Numerous ready-made scripts, add-ons, assets and tutorials are available online.

3.2 Particle Systems As we have seen, complex air pollution data is usually visualised as either highly abstract shapes, such as bar charts, or as clusters of small spheres and cloud-like structures. Bars can be simply modelled as 3D blocks, and animated along the Y axis. Shape clusters are created with particle generators, i.e. small functions that “spout” a stream of forms or particles with specific properties over time. The user can influence parameters such as particle size, form, number, speed, lifetime, variation, colour, animation, and so on. CPU rendering a high number of particles in real-time requires a lot of comput- ing power, but both Unity and Unreal recently tackled that problem with the introduction of GPU rendered particle systems, within the VFX Graph in Unity [Fig. 8] or Niagara in Unreal, respectively. Both systems can render more than 1 million particles. Game engines also contain physics systems such as gravity that can be used to Visualising air pollution datasets with game engines 7 experiment with particle simulations. While they cannot generate scientifically exact data, they can give first impression of approximated behaviours of air pollution in space, such as weight, collision, penetration and so on.

Fig. 8. Unity 3D: VFX Graph particle generator.

3.3 Hardware Air pollution data can be streamed into the engine via a simple and cheap sensor kit containing an ESP and/or a Raspberry Pi. In 2018 the cost for such a set-up is under 50 Euro.

4 Proposal

Visualising air pollution data sets in the context of experimental situations at workshops or makerlabs requires not only a game engine, but also an organisa- tional structure that integrates non-experts. We describe so-called game jams as a model for such an open and experimental event.

4.1 Game Jams Game engines are regularly and productively used in short-term experimental prototyping situations, or game jams: A group of people meets over a specific period of time (48 hours, 2 weekends etc.) and builds an application within that short time frame. The engine usually contains the needed functionality to finish the task, especially in combination with the above-mentioned free assets. While a game jam team should contain participants who have worked with the engine before, by ”visualising” the prototyping process inside the 3D space and the different windows, game engines provide easy access for all participants. Apart from providing a context for experiments, a game jam can also kick off the production of a more polished application for visualising air pollution data 8 Uli Meyer et al. at home. That way, the game engine becomes both a tool for experimental on- the-fly visualisation in 3D space, and for building a permanent visualisation application. The following section outlines two potential use cases for air pollution visualisa- tion in the context of a game jam.

1. Particles By reading air pollution data into a particle system, the actual properties and behaviours of air pollution in space can be visualised. Different visual properties, such as colours or forms, indicate different types of pollution. Experiments with engine physics give an approximation of how pollution parti- cles react to collision with solid or organic objects, to gravity, wind and so on. The particle visualisation works on a static 2D screen. But as the particles fill a three-dimensional space and surround the user (=user camera), other display types may be more efficient. By building the application for an AR display, the user can move the display freely and explore the visualisation in space. The par- ticle simulation can be overlaid and connected to a concrete environment such as a room or a street. Building a simple VR application lets the user experience actual “body” immer- sion in air pollution. How does it feel when you can actually see the pollution that surrounds your body, and even permeates it?

2. Real-time Interaction Streaming the data into the engine in real time, the user can interact directly with the physical environment, for example by removing a source of air pollution, and observing the effect on the visualisation.

5 Conclusion

In this paper we discuss the potential of game engines as an explorative, low- entry tool for visualising complex air pollution data sets. We propose that game engines are suited for non-IT and non-data experts in the context of experimen- tal situations such as workshops or makerlabs. We outline Beritano’s model of data driven visualisation types, which he cate- gorises along an axis from “declarative” to “experimental”, depending on func- tion. As the form of a visualisation follows its function, we then describe different graphic types of air pollution visualisation for different functions, ranging from more abstract to more concrete. We call attention to the specific properties of air pollution, including its invisibility, pervasiveness, and the fact that it permeates organisms, and propose to visualise them with particle simulations. For experimental workshop situations that allow experts and non-experts to work together on air pollution visualisation, we describe the “game jam” model, a pro- totyping method from game design. We outline two use cases for air pollution visualisation in the context of a game jam. Visualising air pollution datasets with game engines 9

References

1. Beritano, S.: Visualisations that really work. (2016). URL https://hbr.org/2016/ 06/visualizations-that-really-work 2. Broscheit, J., Draheim, S., von Luck, K.: How will we breathe tomorrow? In: EVA Copenhagen 2018 - Politics of the Machines - Art and After (2018). URL http://dx.doi.org/10.14236/ewic/EVAC18.10 3. Carbonvisuals: Carbonvisuals website. URL http://www.carbonvisuals.com/all 4. Chen, C.: Information visualisation and virtual environments. Springer Science & Business Media (2013) 5. Dougherty, J.P.: Information technology fluency at a liberal arts college: experience with implementation and assessment. Journal of Computing Sciences in Colleges 18(3), 166–174 (2003) 6. Friendly, M.: A brief history of data visualization. In: Handbook of data visual- ization, pp. 15–56. Springer (2008) 7. GeoscienceAustralia: Data visualisation with the ’oculus rift’ dk2. URL https: //youtu.be/n0r9Q-zsPh 8. Hsu, Y.C., Dille, P., Cross, J., Dias, B., Sargent, R., Nourbakhsh, I.: Community- empowered air quality monitoring system. In: Proceedings of the 2017 CHI Con- ference on Human Factors in Computing Systems, pp. 1607–1619. ACM (2017) 9. Kaidi, Z.: Data visualization. Retrieved 8(22), 2010 (2000) 10. Kim, S., Paulos, E.: inair: measuring and visualizing indoor air quality. In: Pro- ceedings of the 11th international conference on Ubiquitous computing, pp. 81–84. ACM (2009) 11. Kirk, A.: Data visualisation: a handbook for data driven design. Sage (2016) 12. Lamm, N.: Wifi visualised. URL https://www.vice.com/en us/article/9an9m7/ heres-what-wi-fi-would-look-like-if-we-could-see-i 13. Lewis, M., Jacobson, J.: Game engines. Communications of the ACM 45(1), 27 (2002) 14. Lobser, D.: Cosmic sugar (vr experience). URL https://store.steampowered.com/ app/559010/Cosmic Sugar VR/ 15. Lu, W., Ai, T., Zhang, X., He, Y.: An interactive web mapping visualization of urban air quality monitoring data of china. Atmosphere 8(8), 148 (2017) 16. Marks, S., Estevez, J.E., Connor, A.M.: Towards the holodeck: fully immersive virtual reality visualisation of scientific and engineering data. In: Proceedings of the 29th International Conference on Image and Vision Computing New Zealand, pp. 42–47. ACM (2014) 17. PlumeLabs: Plume air report (app). URL https://play.google.com/store/apps/ details?id=com.plumelabs.air&hl=de 18. RandomInternational: Rain installation, barbican cen- tre london. URL https://www.dezeen.com/2012/10/04/ rain-room-by-random-international-at-the-barbican/ 19. San Jos´e,R., P´erez,J.L., Gonz´alez-Barras, R.M.: 3d visualisation of air quality data. In: Proceedings of the 11th International Conference “Reliability and Statis- tics in Transportation and Communication”, Riga (2011) 20. Stone, R.: Serious games: virtual reality’s second coming? Virtual reality 13(1), 1–2 (2009) 21. WeVR: The blue (vr experience). URL https://wevr.com/theblu