Journal of Ambient Intelligence and Smart Environments 4 (2012) 5–27 5 DOI 10.3233/AIS-2011-0129 IOS Press iWorlds: Generating artificial control systems for simulated humans using virtual worlds and intelligent environments Marc Davies* and Vic Callaghan Digital Lifestyles Centre, University of Essex, Colchester, Essex, UK, CO4 3SQ E-mail: {midavi,vic}@essex.ac.uk Abstract. This paper highlights research into using virtual worlds, intelligent environments and mixed reality to create artifi- cial control systems for simulated humans. Following a brief explanation of this project, the beneficial contribution provided by virtual worlds, intelligent environments and mixed reality systems is outlined. Throughout this project a series of experi- mental iWorlds, (virtual and mixed reality intelligent environments) have been used to profile the actions and decisions of humans to generate sets of realistic behaviours and test the project research hypothesis. These worlds consist of a mixed- reality household experiment plus two supporting game-like behaviour experiments, namely ‘Bar-World’ and ‘Battleships’ which were designed to explore and reinforce aspects of the underlying theory. As a core part of this work a description of a new profile generation mechanism is presented, which comprises a genetic programming mechanism that blends real user behaviour data to produce varied artificial profiles for non-player computer characters. In support of this a set of experimental results are presented that reveal the new artificial intelligence mechanism does indeed advance the project aim to create realis- tic simulated humans. A discussion is included that describes the potential benefits that could be gained by incorporating these methodologies into computer games and professional intelligent environment development systems. Keywords: Artificial Intelligence, Genetic Programming, mimicry, mixed reality, virtual worlds 1. Introduction The aim is to augment an existing and widely-used artificial intelligence platform to allow control sys- 1.1. Setting the context tems to be generated for a variety of simulated hu- mans in different scenarios. The new artificial intelli- Creating a human-level artificial intelligence is gence mechanism developed by this project, (UK widely considered to be one of the ultimate destina- Patent Application No. 1012243.0), combines ele- tions of modern computer science research, with ments of Genetic Programming with data gathered countless projects around the world working towards from real people using iWorlds, (virtual and mixed- this goal. However, in the interim, prior to the crea- reality intelligent environments). The intention of the tion of this ultimate artificial intelligence, the stan- new mechanism is to produce collections of realistic dard of what is available frequently falls short of cre- control systems for simulated humans, without extra ating a widely considered replication of a realistic programming being needed to add variability. human-likeness. From a purely software perspective, While a human-level artificial intelligence (or even one key issue promoting this lack of realism is the a replica), would undoubtedly have potential applica- decision-making abilities of an A.I. program. tions across countless fields, this project has chosen This project does not intend to create the ultimate to focus mainly on computer games and related tech- human-level artificial intelligence, concentrating nologies. Many computer games make use of numer- rather on researching a viable interim mechanism. ous computer-controlled humanoid characters, each playing a specific role. Larger games commonly use * Corresponding author. collections of such avatars to create an immersive 1876-1364/12/$27.50 © 2012 – IOS Press and the authors. All rights reserved 6 M. Davies and V. Callaghan / iWorlds: Generating artificial control systems for simulated humans atmosphere, (e.g. the population of a town). It is the are not intended to be exhaustive. In a true NPC con- artificial intelligence system used by these ambient troller different groups of behaviour-sets would be characters that provides the main focus of this project. attached to the program according to scenarios ex- perienced by the person(s) who created the original 1.2. What is human-likeness? profile(s) used to build it. Please note that in this paper the term ‘Personality’ One issue when attempting to create an artificial is used loosely, as a means of describing a collection intelligence capable of producing human-like actions of different behaviour-sets. To this day psychologists is identifying exactly what a majority of people and psychoanalysts are unable to collectively agree would consider realistic. For the purposes of this pro- on what constitutes a personality [24,27]. ject ‘realism’ of simulated humans refers to the ac- tions they choose to perform, (i.e. decisions) rather 1.3. Recording human-like personae than any physical attributes. However, as all indi- viduals possess their own unique set of personality Intelligent environments provide an ideal platform traits, there is no guarantee that a group of people to capture the natural decisions and actions of real would all respond to the same stimuli in an identical people. To gather realistic usage data for use within manner. For individual simulated humans this may the project mechanism a series of physical, virtual not be that problematic, but in cases where multiple and mixed reality intelligent environments, (iWorlds) characters are being used in a group-based scenario, have been created. The aim was to immerse partici- every individual performing the same set of actions pants of evaluations within the test environment as could make an artificial intelligence seem much less much as possible, in an attempt to keep responses to human-like. Of course, there are also some special any stimuli they encountered realistic. cases where it would be considered human-like for 1.3.1. everybody to perform the same action simultaneously, Virtual worlds for example during a fire evacuation [17]. The graphical quality of a virtual world can vary significantly depending on the number of dimensions used and the detail of surface and object textures. 1.2.1. Personality and personae Many virtual worlds representing intelligent envi- In addition to every individual possessing their ronments typically employ a top-down viewing per- own unique set of traits, most humans don’t behave spective with building features and other content be- in the same manner constantly but tend to alternate ing displayed using two dimensional graphics, as their behaviour (i.e. decisions and actions) according shown in Fig. 2 [11,29]. For simple operations this to factors such as their current situation and mood at level of detail can be sufficient, although the lack of a the time. Therefore, in a controller for a mobile non- height dimension can restrict the number of devices player characters (NPC) inhabiting a virtual world, usable within the world. For example, any objects its overall ‘Personality’ could potentially be com- attached to walls or ceilings are displayed at the same prised from a number of smaller behaviour-sets, level as those on the floor or resting on surfaces. This (Personae) created from environment interactions set-up can potentially create issues such as the con- during different situations, (e.g. working in an office, shopping, home-life). Figure 1 shows an example of a potential structure for an NPC controller when ob- served in this manner. The various categories shown Fig. 1. An NPC controller comprised from several behaviour-sets. Fig. 2. Two dimensional world example. M. Davies and V. Callaghan / iWorlds: Generating artificial control systems for simulated humans 7 start to forget they’re taking part in an experiment when interacting with the environment. Of course, creating a virtual environment with immersive prop- erties stands more chance of successfully influencing individuals if the world’s appearance is built to a realistic standard, (e.g. where a chair is easy to iden- tify because it looks like a typical physical chair). Textures used for surfaces and devices should also be considered so nothing appears abnormal or obviously out of place in the virtual environment, with specific objects still being instantly identifiable by individuals. In the case of a Mixed Reality Environment, any Fig. 3. Three dimensional world example. simulated devices that also have a presence in the physical component, (i.e. Duel Reality), should be of cealment of environment features, due to several a similar, (or preferably identical) design to their real overlapping objects distorting the true layout, which counterparts. in turn could lead to inadvertent misinterpretations of Virtual worlds created for intelligent environments the displayed scene by an unfamiliar observer. research projects can be utilised in a mixed reality Naturally, one way to resolve the lack of a height context. Such systems can provide a convenient component is to create a virtual world modelled us- method for interfacing with devices in a physical ing three dimensional graphics, as shown in Fig. 3. environment and/or exhibiting the influence of agents With virtual content being modelled in all three di- operating within the test-space. Other benefits also mensions the world could be observed from a first- emerge from virtual worlds not being subject to the person perspective by users, rather than using a top- same restrictions as physical environments. Temporal down viewpoint [7]. If necessary the three dimen- and climatic features of a virtual world can be cus- sional world could also be viewed from a third- tomised to allow the conditions within an environ- person perspective, with a camera angle set behind a ment to suit the needs of individual experiments [10]. virtual embodiment of the user, allowing the immer- Features of physical devices can be augmented by sive impression of being ‘inside’ the environment to virtual counterparts, revealing relationships or allow- be preserved. This approach is commonly used in ing actions to be performed that would ordinarily be projects such as virtual meeting rooms, where it is impossible, (e.g.