Shift-Life Interactive Art: Mixed-Reality Artificial

Home , Interactive art

Eugene Ch’ng* Shift-Life Interactive Art: NVIDIA Joint-Lab on Mixed Reality University of Nottingham Ningbo Mixed-Reality Artiﬁcial Ecosystem China Simulation 199 Taikang East Road Zhejiang Ningbo China 315100

Dew Harrison Samantha Moore Faculty of Arts Abstract University of Wolverhampton Wulfruna Street This article presents a detailed design, development, and implementation of a Mixed Wolverhampton WV1 1SB Reality Art-Science collaboration project which was exhibited during Darwin’s bicente- United Kingdom nary exhibition at Shrewsbury, England. As an artist-led project the concerns of the artist were paramount, and this article presents Shift-Life as part of an ongoing explora- Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 tion into the parallels between the nonlinear human thinking process and computation using semantic association to link items into ideas, and ideas into holistic concepts. Our art explores perceptions and states of mind as we move our attention between the simulated world of the computer and the real world we inhabit, which means that any viewer engagement is participatory rather than passive. From a Mixed Reality point of view, the lead author intends to explore the convergence of the physical and virtual, therefore the formalization of the Mixed Reality system, focusing on the integration of artiﬁcial life, ecology, physical sensors, and participant interaction through an interface of physical props. It is common for digital media artists to allow viewers to activate a work either through a computer screen via direct keyboard or mouse manipulation, or through immersive means. For ‘‘Shift-Life’’ the artist was concerned with a direct ‘‘relational’’ approach where viewers would intuitively engage with the installation’s everyday objects, and with each other, to fully experience the piece. The Mixed Reality system is mediated via physical environmental sensors, which affect the virtual environment and autonomous agents, which in turn reacts and is expressed as virtual pixels projected onto a physical surface. The tangible hands-on interface proved to be instinctive, attractive, and informative on many levels, delivering a good example of collaboration between the arts and science.

1 Introduction

The Mixed Reality installation titled ‘‘Shift-Life’’ presented in this article was exhibited at the Shift-Time Festival of Ideas (3–12 July 2009) which coin- cided with Charles Darwin’s 200th year celebration. The project was later exhibited at The Wolverhampton Art Gallery and demonstrated at the Gadget Show Live (8–11 April 2010) at NEC Birmingham, UK. The Shift-Life installation was conceived when our lead artist Dew Harrison was invited by the Shrewsbury Museums Services, and funded by Arts Council England, to create a work related to Charles Darwin for the bicentenary of his birth in his birthplace of Shrewsbury. She was one of only 10 artists Presence, Vol. 26, No. 2, Spring 2017, 157–181 doi:10.1162/PRES_a_00291 ª 2017 by the Massachusetts Institute of Technology *Correspondence to [email protected].

Ch’ng et al. 157 158 PRESENCE: VOLUME 26, NUMBER 2

commissioned to make a work in response to Darwin’s engaged in activating the virtual ecosystem and then ‘‘On the Origin of Species’’ for the festival. For this she working with each other to keep it in balance. envisaged a real-world installation that would allow im- From the arts perspective, the project is an attempt to mediate access to a Darwinian virtual world of artificial elucidate Darwin’s thinking using holistic interaction creatures which users could affect via physical inputs. As interfaces. The bug-like creatures in the Mixed Reality a digital media artist with an interest in new technolo- installation are reminiscent of childhood and take the gies, her main focus is on the parallels between the form of jelly sweets and allsorts. The creatures use a human thinking process and computer processing where cartoony aesthetic to make manifest their artificiality; both are nonlinear systems using semantic association to they are clearly hand-drawn characters and not attempt- link items into ideas, and ideas into holistic concepts. ing to present an indexical trace of reality. Their visual Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 She is also curious about perceptions and mind states as roots in confectionary make clear the playful nature of we move our attention between the created virtual world the work and encourage interaction. The green/yellow of the computer and the real world we inhabit. Most of jelly bugs are the plant-eating herbivores and the larger her work is interactive and considers viewer engagement pink crab-like creatures are the carnivores. Darwin was as a participatory, and not as a passive activity. born and spent his childhood in Shrewsbury, where he Harrison uses digital thinking and technologies in her began his observation of natural life forms and started practice, which materializes as installations, interactive his vast collection of beetles. This boyish interest would pieces, looped videos, and online performances. Her last all his life and led to his great insights and contribu- work is concerned with the similarities of memory, mind, tion to science later on as an adult. In response to and associative thinking held between computers and Darwin’s idea, the aim of this ‘‘participatory’’ artwork humans, where she constructs hypermedia systems of was to create an ‘‘alternate’’ biological life as a set of arti- concept-based art by linking art’s ideas into concepts. ficial or virtual organisms that possess similar biological Earlier pieces focused on the complex and enigmatic art processes to their ‘‘real’’ counterparts—growth, repro- works and ideas of Marcel Duchamp, the forefather of duction, competition, and adaptation. The virtual life Conceptual Art, who was immersed in the rapidly devel- exists in a trophic relationship of predator and prey and oping scientific understandings and new technologies of includes sessile (rooted) and vagile (free moving) organ- his time (1887–1968). Duchamp is single-handedly re- isms. Animal intelligence was programmed into the vir- sponsible for the shift in art thinking from the aesthetics tual organisms to allow them survival strategies. The of the art object to the ideas held within that object, and experiment involves the construction of an enhanced for the understanding that the art object needs a viewer Mixed Reality-based environment which is connected to ‘‘complete’’ it, through mental engagement and inter- with wireless sensors with environmental manipulators pretation. Such thinking led Harrison from the Modern for altering the ‘‘climate’’ of the ecosystem. By bringing to the Postmodern and, together with advances in com- virtual ‘‘living’’ creatures into the physical world where puter technology, toward ‘‘interactive’’ art. Harrison they would seemingly respond to audience activity, we began by creating interactive art pieces and now, in line hoped to create a liminal space blurring the perceived with current art theory, builds ‘‘participatory’’ works in virtual and real states to the point where our viewers collaboration with programmers and designers. For might suspend the belief that these life forms were artifi- ‘‘Shift-Life’’ she extended her practice of transposing cial, and thus engage with the work on a deeper level. Duchampian ideas into interactive hypermedia systems Complementary details of the meanings behind the in- by transposing the ‘‘big idea’’ of Charles Darwin into a stallation as an art piece are available (Harrison, Ch’ng, participatory installation. This Mixed Reality piece was Mount, & Moore, 2009). to facilitate a holistic grasp of the necessary adaptation of Here, a Mixed Reality interface is essential in our art- life-forms in their struggle for survival in a volatile envi- work as it encourages intuitive interactivity for visitors ronment. It required the viewers to be physically while the virtual world responds in real time. Participants Ch’ng et al. 159

can physically manipulate a set of tools as everyday objects such as watering cans to radically alter the living conditions of the virtual creatures in their mixed virtual– real habitat. The virtual agents and their habitat are projected onto the installation space to create a sense of reality in the physical space, as if the creatures actually exist in the physical world. Participants alter the living conditions of the environment by pouring liquids to adjust the pH level and humidity of the soil, varying light conditions, and hammering to create earthquakes in the Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 box, altering the conditions and reaction of the real-time agent simulation with immediate effect. The environmental change could be detrimental to the survivability of a certain species, or promote their growth and reproduction. These responses are dependent on the adaptation of their genotype, which describes their survival strategies and trophic networks. Figure 1. A relationship diagram of the Mixed Reality interactive Observation of the instant effect which a participant’s system between participants, sensors, and the environment with the actions had on the creatures and their ecosystem, pro- collection of agents. jected onto the installation space, proffered an understanding of how changes in environmental conditions exhibit and participant reaction. Section 4 is the formal- induce reactive response as a result of their behaviors. ization of our Mixed Reality work, which covers the Participants have direct control on tipping the balance details of the implementation with behavioral mathemat- between survivability and total destruction of the virtual– ics for the artificial life and how they interface with the real ecosystem. Figure 1 is a relationship diagram of the real world via environment sensors. Finally, we discuss Mixed Reality system described here, which illustrates our artwork, the interdisciplinary experience acquired how our artwork connects the relationships between par- during the creation of the Shift-Life artwork, and what ticipants, the interface sensors, and the environment we have learned throughout the project. through its collection of artificial creatures. Participants in the artwork interact with physical sensors via physical 2 Mixed Reality, Art, and Artificial Life props. Sensors activate and send physical (electrical) and virtual (bits) signals to the virtual environment. Virtual The term ‘‘Mixed Reality’’ was first conceived in environment changes cause virtual agents to react, and the seminal paper by Paul Milgram, ‘‘A Taxonomy of pixel information changes (virtual) are fed back to the Mixed Reality Visual Displays’’ (Milgram & Kishino, physical display. From a purely technical perspective, there 1994), yet two decades after the definition of Mixed has been a lack of the developmental aspects of integrating Reality was set, what it includes and its boundaries are interactive artificial life and simulation of ecosystems with still somewhat vague. We felt that the vagueness may be input that affects the simulation in real time from partici- beneficial as it allows a broad range of interpretation, pants via sensors. Our system and the details of its imple- creativity, and expression for the community. A particu- mentation presented here acts as a formalization of a lar interest of ours is how art concerns can involve the Mixed Reality system for the interactive art community, as use of Mixed Reality interfaces (Grasset, Woods, & Bill- well as the mainstream research community. inghurst, 2007; Gwilt, 2009; Marshall, 2009), and the In the next section, we cover the brief background focus and content which drives our art installation is related to the present research. Section 3 documents the artificial life (Langton, 1997), which implies a simulation 160 PRESENCE: VOLUME 26, NUMBER 2

of ecosystems. The simulation of living creatures is of and art practice studies the creation of illusion in com- late a very popular area in virtual environments, yet puter-aided performance (Marshall, 2009). Mixed Real- Mixed Reality systems where simulation of artificial life ity and artworks using mobile devices have also been and ecosystems affected by participants via sensors are studied (Gwilt, 2009). The context of meaningful still relatively unexplored in both art and other disci- encounters with artists and Mixed Reality is crucial and plines. The work here, therefore, aims to formalize and involves the look and feel, multimodal experience and outline the mechanisms involved in this particular strand interaction, and immersion as a subjective emotion, and of work. game play scenarios (Misker & van der Ster, 2010). Mixed Reality generally refers to a suite of technology Shaw, Kenderdine, and Coover (n.d.) Web of Life and that ‘‘overlays our real-world environment with digital, ConFIGURING the Cave allow participants’ physical Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 computer generated object’’ (Costanza, Kunz, & Fjeld, actions to be transposed, altering the aesthetics of the 2009), covering diverse technological subjects from sig- virtual imagery. Ulrike Gabriel’s Breath (1992–1993), nal processing, computer vision, computer graphics, user mentioned in Barker (2009), uses a combination of interfaces, human factors, wearable computing, mobile physical and digital information and processes to gener- computing, information visualization, and the design of ate a particular aesthetic, comingling the physical and displays and sensors (Costanza et al., 2009). Mixed Real- digital to generate aesthetics. While the many case stud- ity is differentiated from VR by the fact that in VR, par- ies of artists using Mixed Reality involve real humans but ticipants are disembodied and completely immersed in static, nonliving computer-generated 2D or 3D objects, an interactive virtual environment involving at least the we believe that the innovative inclusion of ‘‘living’’ vir- sense of sight and contributing to the phenomenon tual agents and concepts from AI will greatly enhance known as presence (Lombard & Ditton, 1997; Luciani, the experience of participants by exploiting more fully Urma, Marlie`re, & Chevrier, 2004). Costanza et al. this digital encounter. (2009) state that in Mixed Reality, the virtual aspects are Artificial Life (Langton, 1997), the core driver of our equally dominant as the physical reality; this is again, in artwork, has been in widespread use in the entertainment contrast with Augmented Reality (Azuma et al., 2001; industry since its potentials were demonstrated by Reyn- Azuma, 1997) where virtual elements are less dominant olds (1987) in the synthesis of emergence in self-organ- in the display. The essence of Mixed Reality is the fusion izing systems such as bird flocks. Karl Sims’ Evolving of the real and the virtual, and this degree of fusion varies Virtual Creatures (1994) is another example. Artificial across projects. Life and Art have been discussed at length (Whitelaw, Mixed Reality is a field with many applications includ- 2004). However, our survey reveals that this interest ing that of the arts world. Artists create works with vari- may not have been appropriately developed within the ous media. Artistic ‘‘technology’’ has progressed rather sparse Mixed Reality community. This is reflected through art-specific media such as paint and canvas to in the lack of literatures formalizing its use. A seminal pa- those designed for other means and so artists now utilize per (Maes, 1995) attempted to contextualize the impor- digital technology in an exploratory manner. Some have tance of believable and entertaining characters in enter- extended the digital medium into the real world via com- tainment with artificial life. Most artificial life characters binations of physical and digital media. This combina- mentioned both in the text and in literature exist purely tion merges the real with the virtual and can be catego- in the virtual environment (e.g., Prophet’s work dis- rized as a form of Mixed Reality media through which cussed at the end of the article)—at the extreme right- artistic ideas can be communicated. Artworks using end of the virtual continuum. Mixed Reality are not new. Grasset et al. (2007) pro- In Artificial Life, the simulation of an environment for vided a survey of artworks in the context of Mixed Real- supporting life is essential. Simulating life implies the ity, exploring artistic innovations via Mixed Reality tech- understanding of the relationship between organisms nology. A deeper study which focuses on Mixed Reality and their environments, such as in Ecological Modeling Ch’ng et al. 161

(Gillman & Hails, 1997). A particular branch of Ecolog- ical Modeling that abstracts away from state variable models is Individual Based Ecology (IBE), generally referred to as Individual Based Model (IBM). Breckling et al. (2005) stated that ‘‘IBMs contrasts with common ecological models which frequently operate on the population level and represent a population as an overall state, thereby specifying rules how the overall state changes.’’ The Individual Based Model aims to develop theories of the adaptive behavior within the content of their popula- Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 tion and environment and seeks to understand the mutual relationship between the adaptive behavior of individual and system-level properties of populations, communities, and ecosystems (Grimm & Railsback, 2013). Individual Based Models are decentralized and is a bottom-up approach (Resnick, 1994) that is particularly suitable for our artwork. Our project employs concepts from Artiﬁcial Life and ecology as drivers of ‘‘life’’ in our artwork for the Shift- Time festival, experimenting with Mixed Reality and applying the idea as an interactive simulation where volt- age and silicon-based life-forms are brought into the physical environment and real-world environmental factors are merged with the virtual via hardware sensors that are partly controlled by participants.

3 The Shift-Life Mixed Reality Art Exhibit

To meet the commissioned development and in Figure 2. Preliminary sketch of the bug-in-a-box ‘‘sweet’’ creatures. order to elicit an entertaining but partial understanding of the Darwinian view of life through our art interactive art, a collection of agent representations, together with an intuitive interface was required, mediated by tangible simple, fun, and to reference Darwin’s childhood in physical activity that participants could interact with. Shrewsbury where he first began his interest in the natu- Here, we explain our approach in the making of Mixed ral world putting small beetles in boxes for observation. Reality art. Sam Moore’s animation style ticked all the boxes. As such, the agent representations were developed as ‘‘bugs-in-a-box’’ ‘‘sweet’’ creatures with a view to mak- 3.1 Agent Representations ing them as approachable as possible to a diverse audi- Our artist specifically chose the animator Sam ence (see Figure 2). The images moved away from a Moore’s work as she did not want the ‘‘creatures’’ to computer-games visual aesthetic of hyper-reality and resemble real-life animals or microbes in any way, toward a deliberately nondigital, nonmicrobe, graphic, instead, preferring them to be flat graphics and non- and comic aesthetic. This positioned the animator’s scientific in their look. The agents also needed to be work in the realms of the overtly rather than covertly 162 PRESENCE: VOLUME 26, NUMBER 2

‘‘made’’ and referenced a clear fantasy world instead of Switching the lamp on would dry out the atmos- attempting photorealism. The creatures were based on phere and enable plants to grow again; however, too pick-and-mix sweets; the carnivore was a liquorice all- much ‘‘sun’’ might be detrimental to the point of sort, the herbivore was a jelly sweet, and the foliage (for wiping out the carnivores entirely! They could, in shade, sustenance, etc.) was based around a selection of fact, become extinct due to their reproduction penny sweets. The creatures were limited to two dimen- method of cloning, unlike the egg-laying herbivores. sions as they were to be observed from above. The com- When this occurred, we had to restart the program pleted representations of the agents and environments to reassure the younger children that they weren’t can be seen in Figure 4. responsible for a complete genocide. Pouring vinegar from a watering can would ‘‘feed’’ Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 the poisonous plants, toxic to all the creatures, but 3.2 The Mixed Reality Art Installation this could be remedied by pouring baking soda liquid and restoring the plant balance, the herbivores’ food. The art installation (see Figure 3) was made to a Hammering on the box sent the carnivores into panic size that can accommodate small groups of people, fami- mode and they would run for cover under the trees. lies, and individuals, accessible to both children and adults. The installation ran across two days at the Shrewsbury event and our observations and interviews 3.3 Participant Reactions showed that all our visitors, young and old, actively con- Once the piece was completed and on show, the tributed as participators by physically moving the box viewers’ slippage between virtual and real states became objects to change the parameters that affected the pro- apparent, even though the setup was basic with plastic jected virtual world. Participants often remained in a watering cans, etc., and the creatures were comic brightly state of reﬂection by passively observing others’ actions, colored line drawings, but the viewers talked about them watching as ‘‘life’’ takes place in the virtual world. The as though they were alive. Their conversations illustrate algorithms implemented were self-sustaining and stable, their understanding of how the animals had to ‘‘adapt to without the need to intervene; as such, it was visually survive’’ the climate changes that were being introduced mesmerizing as noted by our participants. There was into their ecosystem. But more interestingly, we also saw room for contemplation where the virtual world could how this piece worked on a higher level as a signiﬁer for be understood as an analogy for human activity and its human activity—climate change and the Earth: effect on global climate change within our own world. This became evident through our conversations with ‘‘What happens if you change the light, do the crea- participants during the session. The tangible hands-on tures disappear?’’ Someone else requested that the lamp interface proved to be instinctive, attractive, and inform- be brought nearer the box, ‘‘Oh, they’re all dying!’’ ative on many levels, delivering a good example of col- ‘‘So, if the Earth gets too close to the sun, we’ll all laboration between the arts and science. die?’’ ‘‘You’ve completely changed their planet, they’re As our art is ‘‘alive,’’ there were permutations of sce- all dying; it’s not in balance.’’ This meant that they narios documented from the exhibition demonstrating were now extinct ‘‘like the dinosaurs?’’ The participants the many facets of events that could possibly happen, asked us to restart the system to bring the creatures including the following: back into a sustainable environment, which we did. As they poured water, for instance, the humidity ‘‘This could be the Earth that you’ve wiped out, we can’t would alter and some plants may die back; this reboot the Earth!’’ would mean less food for the herbivore green jelly sweet bugs, and consequently fewer bugs for the The participants expressed similar sentiments: ‘‘The pink carnivores to eat. simulation was self-sustaining, stable, and all found the Ch’ng et al. 163 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 3. Participants and the Mixed Reality installation. installation visually mesmerizing.’’ As creators of the sys- vinegar into the watering in order to adjust the pH level tem, we ourselves were occasionally surprised by the emer- of the virtual soil for helping with the growth of certain gent behavior witnessed during the exhibition—simple species of plants. In Figure 3(d), a participant is seen rules can indeed produce complex behaviors, for ‘‘the pointing to a dying creature. Figure 3(e) shows a partici- whole is greater than the sum of its parts’’ (Aristotle). pant hammering the sides of the sandpit box in order to Figure 3 shows various scenarios during Shift-Time create a virtual earthquake. Finally in Figure 3(f), crowds Shift-Life art exhibition. In Figures 3(a) and 3(b), young are seen gathering around the sandpit, mesmerized by participants can be seen adjusting the ‘‘sunlight’’ and the dynamics of the ﬂourishing ecosystem when the adding moisture to the virtual environment to help with environment is appropriately moderated through the the plant growth. In Figure 3(c), participants added Mixed Reality props. 164 PRESENCE: VOLUME 26, NUMBER 2 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 4. Screenshots of the projected virtual environment demonstrating different periods of the simulation.

Figure 4 contains screenshots of the virtual environ- around the group of plants. Some carnivores have died ment when participants interacted with the Mixed Real- due to senescence and from ingesting a herbivore made ity system. Each figure is at a different period of the sim- toxic by ingesting poisonous plants. ulation showing different scenarios. The figure in the top left shows carnivores emerge from the canopies after 4 Implementation of the Mixed an earthquake (activated by participants hitting the Reality Interface ‘‘earthquake’’ sensor with a toy hammer). In the top right are newly hatched herbivores heading toward the Mixed Reality interfaces generally refer to systems clusters of edible plants. At the bottom left are carnivores which augment reality by overlaying real-world environ- in a feeding frenzy. Two herbivores in the same figure ments with computer-generated objects via headsets have just died (multishaded blobs) due to attacks by a or digital displays. Our research experimented with carnivore. At the bottom right are herbivores clustering an inverse of this concept by projecting computer- Ch’ng et al. 165 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 5. A cross-section illustration of the Mixed Reality system.

generated agents and environment into the physical sheet and surrounded by a set of manipulative tools. world and feeding environmental factors (temperature, A projector fixed to the ceiling of the installation projects sunlight, humidity, soil pH, and earthquake) back to the real-time rendering of the virtual agents and its environ- virtual environment. This allows a mixed-mutual rela- ment onto the muslin sheet. The interactive tools are tionship between the real and the virtual—virtual agents made up of a foam-based hammer used for simulating sense our environment while human participants interact earthquakes, two watering cans used for altering the hu- with them. Our approach is particularly suited to rela- midity and pH level of the soil, and a light source for tional artworks involving multiple participants in public altering the luminance to affect both the temperature spaces. The more complex task here is the development and the light/shade of the environment. Figure 5 illus- of an artificial life ecosystem (see Section 4.3), the ‘‘life’’ trates the specifications with a cross-section diagram. of our artwork, which will be the body of technical details covered in this section. We believe that the for- 4.2 Environmental Sensors malization of our Mixed Reality approach will enable more creative approaches in the future when the ecology The sensor system consists of the following of ‘‘living’’ objects becomes necessary. In this context, devices: our lead artist has since expanded her practice toward An attached Oak USB 3-axis accelerometer hidden notions of creating ‘‘living’’ virtual Duchampian art underneath a paper icon attached to the side of the objects endowed with animal behaviors, which might bug box and a toy hammer with which to actuate interrelate to allow new insights into this body of work. the sensor. The output of the sensor was normalized and an exponential weighted moving average 4.1 The Shift-Life Interactive Art applied to smoothen the response from the trans- Installation ducer; The Mixed Reality system is comprised of a 1.2m2 An Oak USB luminosity sensor, actuated by an box filled with polystyrene beads held under a muslin angle-poise lamp; 166 PRESENCE: VOLUME 26, NUMBER 2

An Oak USB relative humidity sensor, protected by vagile—herbivore, carnivore, and the third is sessile—a a Gortex membrane, actuated by pouring water into vegetation type; these are edible herbs, canopies, and the bug box, and poisonous plants. The agents are short-lived (60 seconds A Phidgets USB pH sensor, actuated by pouring minimum and 150 seconds maximum), which is essential vinegar and bicarbonate solutions into the sandpit. for public exhibition where a large population of visitors is expected. The short life span of the creatures allows The sensors were centralized and reported back to the our viewers to see a full scenario of the simulation in a system. Software objects in the environment receive the few minutes of viewing, although people generally stayed signals and adjust the virtual environment accordingly, for half an hour or more. The general behavior of each which affects the virtual agents in a feedback loop. organism is its survivability and the reproduction of A number of optimizations were made to enable the Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 progenies. The survival of the entire ecosystem depends application to appear to be responding immediately; on the balance of the organisms that inhabit the land- that is, unnecessary sensor signals and logs were sup- scape. If the predator outgrows the prey, an imbalance pressed, and each datum reported back to the ecosystem occurs and the system perishes. If the prey outgrows the only if it differed from the previous reading by a edible plants, the food is scarce and the system is at a di- threshold. lemma. If the canopies (large trees) over-reproduce, the predator has little space left to hunt. If the poisonous 4.3 Agents and Environments plants outgrow the vegetation, more and more herbi- This elaborate section details the implementation vores are toxic; this toxic reaction kills the carnivores of the agent-based model and virtual environment, when they eat the herbivores. When the environment is which drives the virtual creatures in our Mixed Reality suitable, the plant species that found its niche reproduces art installation. The subsections describe in detail the be- better (poison plants love high pH levels). The difficulty havioral mathematics and algorithms which are founda- of such an artificial life development is the maintenance tional to the implementation of our Mixed Reality sys- of ecosystem stability. The ‘‘fun’’ part of the exhibit is tem. We believe that by outlining our approach in detail, when users interact with the system by increasing any of our system can be replicated and used by researchers the environmental factors via our sensor network: tem- wishing to extend our work in other application areas. perature, sunlight, humidity, and pH level. For the purpose of standardizing the presentation of the Herbivore and carnivore individuals are characterized agent-based model for our readership, the format of the by: technical presentation of the model in this section fol- 1. Dynamics—speed (hunting, fleeing), eyesight, and lows that of the Individual-Based Modeling (IBMs) field-of-view (FOV). technique in the Ecological Modeling community 2. Physique—age, deterioration (affects the maximum (Grimm et al., 2006). age), energy, hunger threshold, and flesh index. 3. Behavior—impulse, safe distance (security bound- 4.3.1 Purpose. The purpose of the agent-based ary), and feeding distance. model used in our system is to generalize, and to create a 4. Reproduction—number of progenies and sexual collection of extensible living entities which can be inter- maturity. acted with, in real time, via physical inputs. 5. Ecology—adaptability to sunlight, temperature, seismic vibrations, and humidity. 4.3.2 State Variables and Scale. The model we used for our virtual creatures is a predator–prey ecosys- Vegetation is characterized by these traits: their tem simulation and comprises a hierarchical level of indi- energy, resource index, seed count, reproduction age, vidual, metapopulation, community, and environment. dispersal distance, and ecology—adaptability to sunlight, There are three types of agents. The first two are temperature, soil pH, availability of space, and humidity. Ch’ng et al. 167

The population is characterized by size (the number Avoid Canopy and Poison of individuals in a given species) and the community Reproduce when sexual maturity is reached interacts within their trophic network. Populations can Die when the fitness is depleted be controlled (culled) when a threshold is reached. The For Herbivore highest hierarchical level in the model is the environ- Sense the environmental (Temperature, Sunlight, ment, which is discrete, that is, the smallest unit of Humidity, Earthquake) movement is a single pixel. Abiotic factors depend on Change states user interaction and include temperature, sunlight, pH Grow (aging) and die of senescence level, humidity, and seismic vibrations (earthquakes). Avoid Carnivore Environmental factors can be controlled by both on- Change color when toxic plant eaten Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 screen sliders (widgets) and hardware sensors. Table 1 Reproduce when sexual maturity is reached shows an overview of the processes, parameters, and Die when the fitness is depleted individual genotype of the model. The symbols used represent variables in the equations in the Submodels (Sec- 4.3.4 Design Concepts. The concepts presented tion 4.3.7). It is important to note that these variables below are central to simulating an ecosystem of virtual are based on the principal author’s literature review, and creatures which interact amongst themselves and with observation of animal behavior in the wild and in captiv- the environment affected by physical inputs. Since our ity, which were translated into a relative measure during model is a complex system, explicitly presenting these our development. Whilst the variables were meant to concepts will also help with the observation of individual reflect only the behaviors of our imaginary creatures, and collective behaviors during simulation time. they do have semblance to real-world organisms. Emergence: Emergence does not occur at the population level as there are no interactions between the same 4.3.3 Process Overview and Scheduling. Time species of agents. At the community level, certain emer- is discrete steps in the simulation and the aging of the gent phenomena are observable due to the interactions agents proceeds in seconds. Within each second, certain between predator, prey, vegetation (food and protec- processes (simple rules) occur—state changes, growth, tion), and abiotic factors. interaction, adaptation, feeding/fleeing, reproduction Adaptation: Adaptation in the model refers not to the and inheritance, and senescense. Figures 6 and 7 are general adaptation in evolutionary time-scale but is toler- Finite State Machines (FSM) of the Carnivore and ance to biotic and abiotic factors that have already been Herbivore agents. The processes for each autonomous developed. agent are specified in the pseudocode below. Fitness: The fitness of each organism is measured by the Adaptability Measure (Ch’ng, 2007). The creatures For Vegetation are affected by three abiotic factors: temperature, sun- Sense the environment (Temperature, Sunlight, light, and humidity and the plants are affected by six fac- Humidity, pH level, Space) tors: temperature, sunlight, soil, humidity, pH level, and Grow (aging) and die of senescence (Maximum age) space. The temperature, sunlight, humidity, and pH Compete with nearby plants for space level are streamed from electronic sensors installed at Reproduce when sexual maturity is reached strategic locations in the virtual bug box. Soil conditions Die when the fitness is depleted are unchanged and are based on height fields (certain For Carnivore regions are more habitable). The availability of space for Sense the environmental (Temperature, Sunlight, the plants depends on the number of plants growing Humidity, Earthquake) within that space. Fitness-seeking is not modeled. The Change states fitness measure is provided in later sections and is a prod- Grow (aging) and die of senescence uct of the Adaptability Measure. 168 PRESENCE: VOLUME 26, NUMBER 2

Table 1. Overview of Processes, Parameters, and Individual Genotypes of the Model

Global Parameters Symbols & Value(s)

Aging of agents (per second) 1000 ms Simulation time steps (dynamics) Dt¼0.012 s Rendering Frames Per Second (FPS) 12

Abiotic Parameters (depends on where the sensors are placed: indoor/outdoor) Symbols & Value(s)

Temperature 258C

Sunlight 0.6 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 pH 8 Humidity 0.5 Seismic vibrations 0or1

Carnivore Parameters Symbols & Value(s)

Age at start of simulation (seconds) 1

Maximum age Aj¼60 Deterioration Time (a threshold reached where the creature deteriorates) 10 Speed (pixels) s¼2 Thrust e¼2 Hunting Thrust e ¼3 Fleeing Thrust e ¼4 Thrust Limit tlim¼0.5 Friction /¼0.9 Rotation Angle y ¼ 108 Eyesight d¼100 Field of View v¼908 Energy at start of simulation l¼1.0 Energy Loss (when moving) k ¼0.001 Energy Rest Threshold (at a point when agent needs rest) u¼0.2 Energy used in predation k ¼0.6 Hunger threshold (depletion of energy results in hunger) 0.5 Flesh index 1.0 Impulse range (pixels) i ¼ 200 Impulse I¼20 Safe distance (from predation) 80 Feeding distance (prey captured at this distance) 10 Progenies n¼1 Next reproduction age (reproduced only after n seconds) 30 Mature Age Ratio (reproduced when >¼ age*MatureAgeRatio) 0.5 Sunlight (L¼lower range, P¼preferred, U¼upper range) L:0.2, P:0.6, U:0.8 Temperature (8C) (L¼lower range, P¼preferred, U¼upper range) L:4.0, P:25, U:35 Humidity (L¼lower range, P¼preferred, U¼upper range) L:0.3, P:0.5, U:0.8 Seismic vibration (Earthquake) 1 Ch’ng et al. 169

Table 1. (Continued)

Herbivore Parameters Symbols & Value(s)

Age at start of simulation (seconds) 1 Initial size of creature at birth B0¼0.5 Rate of growth g¼5.0 Maximum age A¼100 Deterioration Time (a threshold reached where the creature deteriorates) 10 Speed (pixels) s¼2

Thrust e¼3 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 Hunting Thrust e¼3 Fleeing Thrust e¼8 Thrust Limit tlim ¼ 0.5 Friction /¼0.9 Rotation Angle y¼108 Eyesight d¼250 Field of View v¼908 Energy at start of simulation l¼1.0 Energy Loss (when moving) k ¼ 0.0005 Energy Rest Threshold (at a point when agent needs rest) u¼0.0 Energy used in predation k ¼ 0.5 Hunger threshold (depletion of energy results in hunger) 0.8 Flesh index K¼1.0 Impulse range (pixels) i ¼ 200 Impulse I¼5 Safe distance (from predation) 50 Feeding distance (prey captured at this distance) 3 Progenies n¼5 Next reproduction age (reproduced only after n seconds) 15 Mature Age Ratio (reproduced when >¼ age*MatureAgeRatio) 0.3 Sunlight (L¼lower range, P¼preferred, U¼upper range) L:0.1, P:0.6, U:0.8 Temperature (8C) (L¼lower range, P¼preferred, U¼upper range) L:4.0, P:25, U:35 Humidity (L¼lower range, P¼preferred, U¼upper range) L:0.3, P:0.5, U:0.8 Seismic vibration (Earthquake) 1

Vegetation Parameters Symbols & Value(s)

Age at start of simulation (seconds) 1 Maximum age A¼70 Culling (maximum population) 15 Energy r¼1.0 Resource index K¼1.0 Seed count s¼5 Reproduction age 10 170 PRESENCE: VOLUME 26, NUMBER 2

Table 1. (Continued)

Vegetation Parameters Symbols & Value(s)

Dispersal distance (pixels) D¼130 Sunlight (L¼lower range, P¼preferred, U¼upper range) L:0.1, P:0.5, U:0.8 Temperature (8C) (L¼lower range, P¼preferred, U¼upper range) L:18.0, P:26, U:38 Soil (P¼preferred, U¼upper range) P:0.5, U:0.8 pH (L¼lower range, P¼preferred, U¼upper range) L:5.0, P:8.0, U:10.0 Space (P¼preferred, U¼upper range) P:0.6, U:1.0

Humidity (L¼lower range, P¼preferred, U¼upper range) L:0.3, P:0.5, U:0.7 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Canopy Parameters Symbols & Value(s)

Age at start of simulation (seconds) 1 Maximum age A¼150 Culling (maximum population) 10 Energy r¼1.0 Resource index K¼1.0 Seed count s¼5 Reproduction age 25 Dispersal distance (pixels) D¼250 Sunlight (L¼lower range, P¼preferred, U¼upper range) L:0.1, P:0.8, U:1.0 Temperature (8C) (L¼lower range, P¼preferred, U¼upper range) L:15.0, P:30, U:45 Soil (P¼preferred, U¼upper range) P:0.4, U:0.6 pH (L¼lower range, P¼preferred, U¼upper range) L:1.0, P:7.0, U:10.0 Space (P¼preferred, U¼upper range) P:0.28, U:0.5 Humidity (L¼lower range, P¼preferred, U¼upper range) L:0.3, P:0.5, U:0.7

Poison Plant Parameters Symbols & Value(s)

Age at start of simulation (seconds) 1 Maximum age A¼50 Culling (maximum population) 15 Energy r¼1.0 Resource index K¼1.0 Seed count s¼3 Reproduction age 10 Dispersal distance (pixels) D¼160 Sunlight (L¼lower range, P¼preferred, U¼upper range) L:0.1, P:0.4, U:0.7 Temperature (8C) (L¼lower range, P¼preferred, U¼upper range) L:16.0, P:27, U:33 Soil (P¼preferred, U¼upper range) P:0.5, U:0.8 pH (L¼lower range, P¼preferred, U¼upper range) L:6.0, P:9.0, U:14.0 Space (P¼preferred, U¼upper range) P:0.6, U:1.0 Humidity (L¼lower range, P¼preferred, U¼upper range) L:0.3, P:0.5, U:0.7 Ch’ng et al. 171 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 6. Finite State Machine diagram of the Carnivorous agents.

Figure 7. Finite State Machine diagram of the Herbivorous agents. 172 PRESENCE: VOLUME 26, NUMBER 2

Prediction: Estimation of future consequences of 2009). The simplified model has minor changes to suit agent decisions is not modeled. the difference in sensing of abiotic factors by agents. Sensing: Agents are aware of their own age-related Sensing the Environment: The vegetation agents mechanisms (reproduction, senescence), other agents sense the environment via receptors. The abiotic factors that have direct impact on their survivability, and the abi- that the agents take into account are temperature, sun- otic factors. light, humidity, and soil pH. Biotic factors include the Interaction: Individuals do not interact within their competition for space and the risk of being consumed by species. Carnivores interact with herbivores via preda- Herbivores. Each abiotic factor is measured by the tion. Carnivores avoid canopies, which are protection for Adaptability Measure (AM) (Ch’ng, 2007) and contrib- herbivores. Herbivores interact with all three types of utes to the fitness measure. Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 vegetation. Edible and poison plants provide energy but Fitness Measure and Death: The AM measures four poisonous plants provide an additional chemical which, abiotic factors and a biotic factor to generate the fitness if ingested by carnivores, causes death. for individual agents i at time t (Equation 1) in each sim- Stochasticity: Stochasticity is implemented in the dis- ulation cycle: tribution of seeds (angle and bounded distance), and the f t ¼ u ðÞCTSg t ; (1) impulses of the movement thrusts of the vagile agents. i i i Collectives: Individuals are not grouped into collec- where the output of AM for each fitness related to biotic tives or social groups. or abiotic interactions are computed: ui is the interac- t Observations: Scientific observation of the model itself tion fitness of the pH level of the soil for plant i, Ci is is unnecessary as this is an interactive art simulation. the only biotic local interaction fitness of the current t t condition ci (Equation 3) at time t, Ti is the interaction t 4.3.5 Initialization. At the start of simulation, all fitness of the plant related to the temperature, Si is the t agents are randomly distributed in the landscape using fitness affected by the sunlight, and gi is the fitness of the default parameters listed in Table 1. Similarly, abiotic the plant in the current humidity. The interaction of the factors are initialized and read from the hardware sen- factors is a logical way for deciding the fitness of the sors, which depends on the current temperature at the plant. The variable Resource Index r, defined as the stor- place of installation. The environment varies depending age of energy, is decremented j unit (see Table 1) in the on where the sensors are outdoors, or within an enclosed condition in Equation 2. Death occurs when tþDt space. qi 0:0, qtþDt ¼ qt j f t 0:0 : (2) 4.3.6 Input. Abiotic parameters are directly read i i i i from the hardware sensors. The raw data is polled and processed every 0.012 ms. This corresponds to the simu- Competition: The collective occupation of the space lation cycle of the agents, ensuring a continuous flow of used by the competing plants contributes to the accumu- t abiotic information to the biotic components. lated space ci at time t for the plant i in Equation 3. Competition for space is defined as an interaction. 4.3.7 Submodels. This section explains in detail A plant interacts with its neighbor in the condition, all submodels and parameterization representing the " rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi Xn 2 processes listed within Section 4.3. The section covers t t t 2 t t c ¼ Pi O u þ O u the mathematical structure of the vegetation behavior i x x y y i¼1 # and the dynamics of the vagile agents. hi t t t t t t Osize þ usize < 0 Osize usize Oage uage ; 4.3.8 Vegetation Behavior. Vegetation behavior t : is a simplified version of an agent-based model (Ch’ng, for 0 ci 1 (3) Ch’ng et al. 173

where n is the number of competing plants, and Pi is the distance of the parent plant (see Table 1); finally b is a effective space used by a single plant. Pi ¼ 0.05 if the random angle in the interval [0,360]. undergrowth species (vegetable or poison) competes against its own species, P ¼ 0.12 if the undergrowth com- i 4.3.9 Carnivore and Herbivore Behavior. petes against a canopy, and P ¼ 0.02 among canopy com- i Sensing the Environment: The vagile agents sense the petition. The differences in P adjusts the space so that the i environment via receptors. The abiotic factors that the canopies are not too crowded together. Ot is the position x;y agents take into account are temperature, sunlight, and of the competitor and ut is the position of the source x;y humidity. Each abiotic factor is measured by the Adapta- plant at time t. Ot and ut , Ot ,andut are respectively size size age age bility Measure (AM) (Ch’ng, 2007) and contributes to the diameter and the age of the two competing plants. the fitness measure. Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 Growth and Reproduction: Growth and reproduction Fitness Measure and Death: The AM measures each uses time as a measurement. Growth is by aging; plants environmental factor, which is combined with other fac- age every second and senescense occurs when the maxi- tors to generate the fitness f v for each individual agent j mum age is reached. Reproduction for Canopy and Veg- j (Equation 6) in each simulation cycle: etable depends on the parameter Reproduction Age and t t the number of seeds s. Poison plants reproduce quicker fj ¼ ðÞTSg j ; (6) on the high pH level of the soil. The pseudocode below where the output of AM for each fitness related to abiotic describes the process; the parameters for the variables t interactions is computed, Tj is the interaction fitness of below are listed in Table 1. t agent j at time t in the current temperature, Sj refers to the interaction fitness with sunlight, and gt is the fitness For Canopy and Vegetable j in the current humidity. When f 0.0, the agent dies. nextReproductionAge ¼ lastReproductionAge þ j Growth and Reproduction: Growth and reproduction reproductionAge uses time as a measurement. Growth is defined by IF age nextReproductionAge aging—the agent ages every second and senescense Reproduce n seeds occurs when the maximum age is reached. Graphically, lastReproductionAge ¼ age the size of the Herbivore agents has the following ratio. For Poison Carnivore size does not change. nextReproductionAge ¼ lastReproductionAge þ reproductionAge tþDt 0 1 1j ¼ 1j þ ; (7) IF age nextReproductionAge – acidity (f) g 1A14t 1 þ e j Reproduce s seeds lastReproductionAge ¼ age where g is a constant, ( g ¼ 5.0) is the Rate of Growth to

The variable acidity is conditional: reach full size, it is the growth spurt, Aj is the Maximum 0 f¼10 IF pH 0.9, f¼5 IF 0.6 pH < 0.9, ELSE f¼0 Age of the agent, and 1j is the Initial Size of creature at birth. Reproduction distance and direction is stochastic: Reproduction depends on the parameter Mature Age Ratio, Reproduction Age, and the number of Progenies x ¼ x þ D X cos b: (4) s i i n. The progenies appear from below the agents when they are reproduced. The pseudocode below describes y ¼ y þ D X sin b; (5) s i i the process; the parameters for the variables below are where xs and ys are, respectively, the x and y position of listed in Table 1. the seed, xi and yi are, respectively, the x and y position of the parent plant, X is a function from R to [0,1] and For Carnivore and Herbivore is a stochastic variable. Di is the maximum dispersal IF age age*MatureAgeRatio 174 PRESENCE: VOLUME 26, NUMBER 2

nextReproductionAge ¼ lastReproductionAge þ t t t t t reproductionAge vjk ¼xjksin xj þ yjk cos xj ; (13) IF age nextReproductionAge where, Reproduce n number of progenies t t t xjk ¼ xk xj : (14) lastReproductionAge ¼ age t t t yjk ¼ yk yj : (15) 4.3.10 Behaviors and Dynamics of Carnivores and Herbivores. These behaviors are our approach in The visibility of the agent from agent j to k and their defining the movement dynamics and behaviors of our position x and y at time t, rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi virtual creatures, such as chase and evade, roaming and 2 2 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 t t t t t scavenging, and energy gain and loss. These are founda- djk ¼ xj xk þ yj yk : (16) tional formulae, essential to the implementation of our simulation, which bring the virtual creatures to life. The The target agents are visible to the hunting agents at 908 formulae can be extended for application in other field of view, research. #j ¼ vjk > 0 ; (17) Movement when Targeting Another Agent: During one simulation time step, the steering behavior of the where Wj is the decision to hunt (move to) the target. agent position updates with the speed, thrust (to propel Movement during the ROAM State: When in the the body forward), and heading, roaming state, both Carnivore and Herbivore use their impulses to navigate. The impulse to turn either right or tþDr t t t left and the thrust to move forward in short bursts xj ¼ xj þ rj sj cos xj ; (8) depend on Equations 18 and 19, tþDt t t t þhj if Xtj < Ij yj ¼ yj þ rj sj sin xj ; (9) t xj ¼ (18) hi hj if Xtj > Ij : where xtþDr ytþDt is the vector of agent j with speed j j t sj (length of the vector) and Thrust sj (changes only ej if Xtj < Ij sj ¼ (19) during the roaming state, see Equation 20) at time tþDt. 0 otherwise; The heading xt (orientation of the vector) depends on j where o is the heading of the agent j with steering angle the position of the target agent j, j y at time t, the thrust t as the decision to propel the j j þhj if ujk > 1 body forward with force ej, X is a function from R to xt j ¼ (10) [0,1] and is a stochastic variable, l is the Impulse Range, hj if ujk < 1; j and Ij is the Impulse of the agent (see Table 1). During where yj is the agent heading. Avoidance behavior is the roaming state a friction is applied to the thrust for defined below by simply swapping the sign associated the simulation time step tþDt, with the heading, ( st / if st / > slim tþDt j j j j j h if u > 1 sj ¼ (20) t j jk slim otherwise; xj ¼ (11) j þhj if ujk < 1: t lim where s is the Thrust in the Thrust Limit s and /j is ut and vt are the visibility (Eyesight) from agent j to tar- j j jk jk the Friction applied to the force. get agent k with the distance between the agents in the x Hiding and Fleeing (Earthquake or Targeted by and y component xt and yt at time t, jk jk Predator): Carnivore goes into the hiding state when earthquakes occur and Herbivore goes into the fleeing ut ¼ xt cos xt þ yt sin xt : (12) jk jk j jk j state when targeted by a predator: Ch’ng et al. 175

Table 2. Energy Gain and Loss for Both Herbivore and Carnivore

States k Loss (Herbivore) k Loss (Carnivore) þk Gain (Herbivore) þk Gain (Carnivore)

ROAM 0.0001 0.001 - - HUNT 0.0001 0.001 - - FLEE 0.001 0.001 - - CHASE - 0.001 - - SEARCH 0.0001 - - - HIDE 0.0001 0.001 - -

REST - - 0.1 0.01 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

IF earthquake sensed THEN find the closest Canopy Submodels section are covered here. Figure 8 shows an Hide under the Canopy until earthquake magnitude overview of the classes. Figures 9 and 10 illustrate the subsides properties and methods of the class relationships IF targeted by predator THEN find the closest Canopy between the agents and the environment. These impor- Hide under the Canopy until predator is gone tant figures are included here so that the Mixed Reality community can reproduce the work. Energy Gain and Loss: Energy is gained via predation, The Main class acts as the controller for the entire sys- and food intake and energy loss occur during the move- tem—setting up the user interface, initializing the agent ments. The use of energy depends on the efficiency of and environment objects, updating, and managing and each species with the gain and loss, rendering the graphics Sprite layers associated with each agents (UILayer, SoilLayer, HerbivoreLayer, Carnivore- ktþDt ¼ kt þ j; (21) j j Layer, CanopyLayer, VegetableLayer, and PoisonLayer). tþDt Note that the Sprite layers are to make sure that the ren- where kj is the energy of the agent i at time tþDt and k defined in Table 2 is the amount of energy gained or dering order is correct. For example, the SoilLayer lost. When Carnivore acquires Herbivore as food, the should be the lowest layer (rendered first as a back- Flesh Index (see Table 1) K is added to the energy of the ground) while the CanopyLayer should be at the top Carnivore. Herbivore acquires its energy from the (rendered last). Resource Index of the plants. The agent rests (stops activ- The subsequent text explains the Environment t and Agent classes in Figures 9 and 10 in more detail. ities) in the condition kj < tj where tj is the Energy Rest Threshold (see Table 1). For the purpose of more effectively visualizing the UML diagrams, the Flash event methods (Event, 4.3.11 The Object-Oriented Implementation Keyboard, Mouse, Timer, etc.) are not shown in the Details. During the preparation for the exhibition, it diagrams. was decided that Adobe Flash Action Script 3.0 (Object- The Environment class manages the connection with Orientation) should be used for the development for data input from the hardware sensors. Each variable in practical reasons (i.e., portability to the Web); however, this list is linked to the hardware sensors—sunlight, tem- the general mathematics algorithms can be applied to perature, humidity, pH, and isTremor. The isTremor any object-oriented languages, such as a separate com- requires other related variables to work. The timeHan- plex systems built for large-scale simulation in Java/ dler() and updateLiveStream() functions manage the Cþþ (Ch’ng, 2013), or a simplified version of plant dis- isTremor variable. timeHandler() is updated every sec- tribution for virtual environments and visualization ond and updateLiveStream() reads the sensors every (Ch’ng, 2011). Details that are not covered in the 0.012 ms. The pseudocode describes the process: 176 PRESENCE: VOLUME 26, NUMBER 2 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 8. UML class diagram depicting the class relationships. timeHandler() variables—plants, prey, predators. Each agent has a unique if isTremor ¼ True identification index and the agents identify their food tar- tremorCountþþ get, competitors, and protection (Canopy) via these varia- if tremorCount > 5 bles—competitorIndex, foodTarget, plantIndex, and cov- isTremor ¼ false erIndex. The index of a particular target is obtained from tremorCount ¼ 0 the functions ProximityTest() and FindPreyType().The liveTremor ¼ 0.1 Carnivore and Herbivore classes use the AnimalInfo class liveTremorPast ¼ 0.1 to access their genotype via an XML file. These vagile agents are differentiated from the plant agents by the updateLiveStream() AnimalState, movement variables and methods, and the tremorMagnitude ¼ liveTremor – liveTremorPast lesser number of abiotic factors that affect them. if tremoreMagnitude > ¼ 0.1 liveTremorPast ¼ liveTremor 5 Discussion isTremor ¼ true This section is a reflection of our artwork after hav- The three plant agents—Canopy, Poison, and Vegeta- ing successfully collaborated in this interdisciplinary pro- ble—use the PlantInfo class for accessing the genotype ject, in having designed, implemented, and deployed our parameters stored in an XML file. Each object derived exhibit in multiple venues, and interacted with partici- from the classes senses the live stream from the abiotic pants representing a diverse range of demographics. sensors from the Environment object. The timeHan- In this project, we explored a particular interest on dler() method manages the age increment of the plant how an artwork can better engage with audiences via a and senescence (when age > maxAge). The update() robust Mixed Reality interface, and as a positive conse- method manages the rendering of the frames (see Figure quence, investigated an innovative approach in converg- 9), calls the fitness() methods, and checks for scheduled ing the real and the virtual in a true, Mixed Reality sys- reproduction. tem. The content of our artwork was Darwinian The LifeProcesses class is used by all agents for meas- adaptation for survival. For this participatory art, we uring its adaptability to biotic and abiotic factors. All ‘‘mixed’’ reality by simulating virtual life which can be agents are made aware of other species via the Collection interacted with via actual physical human activity, Ch’ng et al. 177 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 9. UML class diagram depicting the detailed class relationships among vegetation and the environment. 178 PRESENCE: VOLUME 26, NUMBER 2 Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021

Figure 10. UML class diagram depicting the detailed class relationships among carnivores, herbivores, and the environment. Ch’ng et al. 179

converging the virtual and the physical in an innovative ment, they would chase or evade each other, eat, grow, Mixed Reality system which we have formalized. We mate, and die. However, the creatures were completely accomplished a truly mixed reality system by integrating immersed in a virtual world. In contrast, our artwork has real-time display, artificial life, ecological simulation, and a strong basis in Darwinian principles, brought forth real-time sensors, and physical props in a participatory through a Duchampian understanding of art, while Squid artwork. Life itself and the shaping of life by nature’s Soup and Prophet’s artworks were largely concerned with forces have always fascinated humankind. A mixed reality novelty and entertainment. The Duchampian under- system requires such an aspect of work; our experience standing shifts from the aesthetics of an art object, to the with the Shift-Life exhibit suggests that virtual life can ideas held within that object, and that an art object needs lengthen viewer participation, fascinate an audience, pro- a viewer to ‘‘complete’’ it through mental engagement Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 mote discussions, and enhance content sustainability. and understanding, and such is our Mixed Reality art- Our unique Shift-Life approach was the propitious pro- work, an attempt at elucidating Darwin’s thinking using vision by Darwin’s Bicentenary Shift-Time Festival, and an interactive interface. By transposing the ‘‘big idea’’ of the eventful gathering of two faculties, allowing us to Darwin into a participatory exhibit, we created a liminal explore, and consequently experiment with such an art- space, blurring the perceived virtual and real states to the work, which we believe extends the concept of Mixed point where our viewers might suspend the belief that Reality to another level—for what is true Mixed Reality, these life forms were artificial, and thus engage mentally if one of the worlds, which lies on the one side of the with our work at a deeper level. interface, is lifeless? In our collaboration, in the first stage of development, Traditionally, digital media artists tend to engage with there were questions on the adaptation and natural selec- their viewers either through a computer screen or projec- tion of our creatures which we hope to address further tion and direct keyboard or mouse manipulation, or for future projects. These were questions on the evolu- through the physical triggering of sensors to activate tion of our creatures and how we could appropriately their work. For Shift-Life we were concerned with a represent evolution within a limited time allowance for more direct ‘‘relational’’ and participatory approach participatory exhibits, and in terms of the feasibility of where viewers would both intuitively engage with the project-hours. The Shift-Life project has a limited time installation’s everyday objects, and also with each other span from funding to exhibition, and, due to constraints to more fully experience the piece as a Mixed Reality art- of time, we were unable to develop algorithms that sim- work. Our work is in contrast with contemporary artists, ulate evolution (and for good reasons as participant such as the Squid Soup collective and Jane Prophet, both viewing times were not long enough to witness evolu- of whom are digital media artists with an interest in com- tion anyway, which requires significant generations of puter games and play, as evident in their works which are reproductions and mutations to occur, and within evolu- similar to computer games such as Framsticks: 3D simu- tionary timescales). Nevertheless, we believe that we lation and evolution and MIT’s Creatures. Squid Soup have achieved important aspects of Darwin’s observation has done works which project light shapes of insect-like in our simulation. creatures into a box of sand where they could run over We have also acquired experience in Mixed Reality your hands, these having no AI programmed into them. exhibits—the constant flow of visitors activating the sen- TechnoSphere created (1995–2002) by Jane Prophet/ sors by hammering, pouring liquids, etc., indicated that Gordon Selley was an online virtual world for users to for longer sessions with more participants, we will need build creatures and release them into a real-time 3D sim- to design more robust methods for protecting the sen- ulation environment where they would interact. The sors and a more efficient way of dealing with the excess program was relaunched as TechnoSphere III in 2012 liquids collected underneath the display. Our future pro- with AI programming for simple evolution; the creatures ject should result in a more sophisticated Mixed Reality responded as individuals to each other and the environ- system, building from the foundational work described 180 PRESENCE: VOLUME 26, NUMBER 2

here. The new system should act as a platform for deeper were important here where I was to elicit scientific prin- thoughts on the understanding of interactive and partici- ciples to a nonscience audience. The idea was to make patory art practice. Our further exploration of both the hard science and complex technical details disappear for conceptual and systemic aspect of Mixed Reality should the participants, to allow for Ihdeian ‘‘embodiment yield further insights and techniques for the community relations’’ (Ihde, 1990) to take place where the percep- interested in our work. Our creatures, programmed with tion of technology ‘‘withdraws’’ into the body and the simple rules, have produced a multiplicity of behaviors human experience through technology which is barely during the exhibitions, some of which were surprising noticed by the users. Getting collaborators with the right and unpredictable. This phenomenon also expresses the skills was an important part of this project, each accom- participatory art to a higher level—life is sustainable and, plishing their role in developing the installation in tune Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 therefore, the observation of our artwork is also sustain- with my overall vision for the piece. I was delighted with able. In the future, we hope to program and animate the the end product which chimed so well with my original creatures not only to rapidly evolve behavior in accord- idea. This has led me to reconsider my approach to prac- ance with their volatile world, but to also have the crea- tice; new works will now move away from screen-based tures emerge with unpredictable social patterns from the projections and advance the inter-relationship between a individually programmed behaviors, and thus bring the virtual Duchampian world and the viewers’ world Darwinian principle to the full. through Mixed Reality interfaces and artificial life From observing members of the public engaging with simulation. the art installation, it was pleasing to note that they needed no encouragement to do so, in that they found Acknowledgments the implements playful and used them intuitively as intended. The simple creature aesthetics were therefore Our thanks for funding and support go to the Arts Council the right approach to delivering this sophisticated artifi- England, the Shrewsbury Museums Service, and the Shift-Time cial life system to a nonscience audience, particularly Festival organizers. where the simple 2D child-focused creatures and plants were concerned. On replaying the videos captured dur- References ing the event, you can clearly hear the participants talk- ing about the piece and their growing understanding of Azuma, R., Baillot, Y., Behringer, R., Feiner, S., Julier, S., & Darwinian adaptation for survival while also grasping the MacIntyre, B. (2001). Recent advances in augmented reality. analogy to climate change affecting our actual world. Computer Graphics and Applications, IEEE, 21(6), 34–47. They worked easily together in a relational way, often Azuma, R. T. (1997). A survey of augmented reality. Presence: with complete strangers, to either nurture the creatures Teleoperators and Virtual Environments, 6(4), 355–385. or disturb their environment. It is evident that they had Barker, T. (2009). Process and (mixed) reality: A process phi- experienced a meaningful encounter with the artwork on losophy for interaction in mixed reality environments. IEEE show and had obviously enjoyed this experience: ‘‘That’s International Symposium on Mixed and Augmented Reality- well good! Where did you get it from? ...you made it? Arts, Media and Humanities, 17–23. Breckling, B., Mu¨ller, F., Reuter, H., Ho¨lker, F., & Fra¨nzle, WOW!’’ O. (2005). Emergent properties in individual-based ecologi- Finally, we present our lead artist Dew Harrison’s cal models—Introducing case studies in an ecosystem reflection as a summary conclusion to our article: research context. Ecological Modelling, 186(4), 376–388. Ch’ng, E. (2007). Modelling the adaptability of biological sys- The collaboration with computer science and an ani- tems. The Open Cybernetics and Systemics Journal, 1, 13–20. mator allowed me to explore a means of engaging view- Ch’ng, E. (2009). An artificial life-based vegetation modelling ers in my artwork in an immediate and more fulfilling approach for biodiversity research. In R. Chiong (Ed.), way. The aesthetics of the piece, the look and feel of it, Nature-inspired informatics for intelligent applications and Ch’ng et al. 181

knowledge discovery: Implications in business, science and engi- Lombard, M., & Ditton, T. (1997). At the heart of it all: The neering (pp. 68–118). Hershey, PA.: IGI Global. concept of presence. Journal of Computer-Mediated Commu- Ch’ng, E. (2011). Spatially realistic positioning of plants for nication, 3(2), 0. virtual environments: Simple biotic and abiotic interaction Luciani, A., Urma, D., Marlie`re, S., & Chevrier, J. (2004). for populating terrains. IEEE Computer Graphics and Appli- PRESENCE: The sense of believability of inaccessible cations, 99. worlds. Computers & Graphics, 28(4), 509–517. Ch’ng, E. (2013). Model resolution in complex systems simu- Maes, P. (1995). Artiﬁcial life meets entertainment: Lifelike au- lation: Agent preferences, behavior, dynamics and n-tiered tonomous agents. Communications of the ACM, 38(11), networks. SIMULATION, 89(5), 635–659. 108–114. Costanza, E., Kunz, A., & Fjeld, M. (2009). Mixed reality: Marshall, J. (2009). Creating illusion in computer aided per-

A survey. Berlin: Springer. formance. Doctoral dissertation, University of Nottingham. Downloaded from http://direct.mit.edu/pvar/article-pdf/26/2/157/1836477/pres_a_00291.pdf by guest on 27 September 2021 Gillman, M., & Hails, R. (1997). An introduction to ecological Milgram, P., & Kishino, F. (1994). A taxonomy of mixed real- modelling: Putting practice into theory. Oxford: Blackwell ity visual displays. IEICE Transactions on Information and Science. Systems, 77(12), 1321–1329. Grasset, R., Woods, E., & Billinghurst, M. (2007). Art and Misker, J. M. V., & van der Ster, J. (2010). Authoring immer- mixed reality: New technology for seamless merging sive mixed reality experiences. In The engineering of mixed between virtual and real. Proceedings of DAC. reality systems (pp. 275–291). Berlin: Springer. Grimm, V., Berger, U., Bastiansen, F., Eliassen, S., Ginot, V., Resnick, M. (1994). Turtles, termites, and traffic jams: Explora- Giske, J., ...Huse, G. (2006). A standard protocol for tions in massively parallel microworlds. Cambridge, MA: MIT describing individual-based and agent-based models. Ecologi- Press. cal Modelling, 198(1), 115–126. Reynolds, C. W. (1987). Flocks, herds and schools: A distrib- Grimm, V., & Railsback, S. F. (2013). Individual-based model- uted behavioral model. ACM SIGGRAPH Computer ing and ecology. Princeton, NJ: Princeton University Press. Graphics, 21(4), 25–34. Gwilt, I. (2009). Augmented reality and mobile art. In Hand- Shaw, J., Kenderdine, S., & Coover, R. (2011). Re-place: book of multimedia for digital entertainment and arts (pp. The embodiment of virtual space. In T. Bartscherer (Ed.), 593–599). Berlin: Springer. Switching codes: Thinking through digital technology in the Harrison, D., Ch’ng, E., Mount, S., & Moore, S. (2009). humanities and the arts. Chicago: The University of Experiencing the Big Idea. 8th International Conference on Chicago Press. Digital Arts and Culture, 199–204. Sims, K. (1994). Evolving virtual creatures. Proceedings of the Ihde, D. (1990). The Indiana series in the philosophy of technol- 21st Annual Conference on Computer Graphics and Interac- ogy. Bloomington, IN: Indiana University Press. tive Techniques, 15–22. Langton, C. G. (1997). Artificial life: An overview. Cam- Whitelaw, M. (2004). Metacreation: Art and artificial life. bridge, MA: MIT Press. Cambridge, MA: MIT Press.