The International Journal of Virtual Reality , 2017, 17(03): pp1-pp14 1 An Agent-Aware Computing Based Mixed Reality Information System Charles Z. Liu 1;2;3 and Manolya Kavakli 1;2 1 VISOR, Faculty of Science and Engineering, Macquaire University 2 Department of Computing, Macquaire University 3 PSI Team, SmartSys Workgroup Abstract - In this paper, we present an agent- AR consists of more reality than artificially modeled virtu- aware computing based collaborative information sys- ality. Many AR systems are more widely used than immer- tem scheme for MR applications. The HCI issues in VR, sive VR systems due to their low cost and easily accessible AR, and MR, have been discussed firstly. Besides the characteristics (Ong and Nee 2013; Lee and Akin 2011; equipment, the lack of understanding of users also ac- Kaufmann and Schmalstieg 2003). However, when im- counts for a significant bottleneck of improving user ex- mersive interaction is needed, a mixture of virtual and real perience and immersion during the interaction. Aiming worlds supported with equally physical and augmented ele- at the issues, an agent-aware computing based scheme ments is required. is proposed and implemented. The corresponding sys- Mixed reality (MR) systems (Milgram and Kishino 1994; tematic scheme is presented in the views of functional- Tamura, Yamamoto, and Katayama 2001; Costanza, Kunz, ity, modules and system workflow and discussed in de- and Fjeld 2009) were proposed to perform merging of real, sign, implementation, and system synthesis. By means with a cyber world. Due to this physical and digital coex- of the human-aware computing, system-aware comput- istence, a mixed reality system allows individuals to expe- ing and human-system aware computing, issues such as rience a high level of immersion. Reproduced multimedia mixed reality fusion, QoE-QoS management, confiden- metaphors (related to senses of sight, hearing, smell, taste tiality, and security, are addressed by the applying agent- and touch), in MR encompass both augmented reality and aware computing based scheme into an implementation augmented virtuality. This type of systems provide possible of the collaborative information system for mixed real- improvements for high-level of interaction, such as memory ity. Related modules along with their function and test functioning (Parsons and Rizzo 2008), attention enhance- results are given and discussed. ment (Cho et al. 2002) and behavior therapy (Klinger et al. Index Terms - mixed reality fusion; collaborative infor- 2005). mation processing; agent-aware computing based design To improve the performance of MR systems for a high- level of interaction, modern computer systems have pro- Introduction vided many auxiliary functionalities. These perform anal- ysis on data and visualization with marked characteris- Virtual Reality (VR) refers to an alternate world filled with tics (Menache 2000; Bickel et al. 2007), and require some simulated environments and objects. Computer-generated sensors and actuators (as in (Gulrez and Kavakli 2007)) for images allow the end user to interact with the virtual space. such interaction. However, due to the lack of understanding The virtual reality presents the user the experience of a mix- of users’ intention, those devices play roles similar to key- ture of reality and virtuality (Coates 1992; Greenbaum 1992; boards and mice. The MR systems perform mostly passive Steuer 1992). One of the important issues in VR is im- interaction. Passive interaction leads to a reactive approach mersive interaction (Bowman and McMahan 2007; Psotka in current HCI practices (Stephanidis 2001), and this reac- 1995). tive interaction may reduce the quality of user experience VR can be divided into 3 kinds (see table 1): textual VR, and the usability of the system, particularly for the scenario which refers to interaction without visualization; desktop that user needs the system to assist in task completion. VR, interaction with visualization; immersive VR, interac- The goal of this paper is to develop a framework for an tion with high level of immersion(Burdea Grigore and Coif- intelligent HCI (iHCI) system using a collaborative infor- fet 1994). mation system for MR in (Liu and Kavakli submitted; ) and Augmented Reality (AR) refers to an experience in a di- allow the system to understand users intention during the in- rect or indirect view of a physical, real-world environment. teraction. This requires the systems to be able to understand An AR system provides a virtual layer of information with the subject and context in human-computer interaction. This computer-generated images in reality. Compared to VR, will lead to a better user experience and information security using context-aware computing (Hong and Landay 2001; E-mail: [email protected] Gay 2009). The context refers to the environment, condi- The International Journal of Virtual Reality , 2017, 17(03): pp1-pp14 2 Table 1: Main Types of Virtual Reality Types of VR Visualization Interaction Immersion p p Textual VRp - p Desktop VR p p p- Immersive VR tions and surroundings that involve in the subject system experience assessment. It is a challenge for a machine to un- and its computing. Context-aware computing here refers to derstand abstract concepts like user experience. computing operations of computers to simulate awareness Agent-based models (ABM (Getchell 2008; Helbing of context by sensing the physical environment, and adapt- 2012)) are micro-scale models that simulate the simultane- ing systems’ behavior accordingly. ous operations and interactions of multiple agents to cre- ate or predict complex phenomenon. Simple behaviorial Related Works rules generate complex behavior. Thus, agents may ex- perience learning and adaptation. Knowledge systems can The applications of computational intelligence in virtual en- help HCI to acquire human sense of interaction and expe- vironments can be seen as a convergence of smart system rience. Many works related to knowledge-based HCI fo- and HCI systems. How to develop an intelligent HCI (iHCI) cused on knowledge translation and knowledge system im- system is a key to this convergence. plementation for system operations (e.g. (Seo et al. 2017; Agent-based design can be used to implement the context Alverson et al. 2015; Levac et al. 2013; Picinali et al. 2014)). aware iHCI systems. There are many studies (e.g. (Li 2015; The knowledge in these works is mainly used as a logic of Ruhland et al. 2014; Kinateder et al. 2014; Luck and Aylett rules and data. Efficacy of the knowledge-based instruction 2000; Rickel and Johnson 1999)) on intelligent system de- was the primary criteria of evaluation. However, an iHCI sign. Issues related to intelligent VR systems, are knowl- system must take the human perception and cognition into edge representation, autonomous agents, and agent cou- account to simulate the real response by a human rather than pling. They showed perform some capabilities for intelli- a logic link between the rules and the knowledge base. Only gence, such as perception, cognition, recognition and in- with this process can human experience be modeled and struction. However, those intelligent interactions were based computed. However, human perception is abstract. One of on the results of machine states influenced by the users’ op- its essential characteristics is uncertainty. Subjective aware- eration. Even though some complex factors (such as social ness is hard to quantify, while computation needs to be spe- influence) have been considered, the behavior of agents still cific and accurate for the processing of interactive data. How are controlled by a certain dynamic system model rather than to solve this contradiction is a challenge for intelligent HCI the context. system. Networking applications mainly have focused on commu- Moreover, when considering wide area mixed reality ap- nication, traffic flow and implementation of the interactive plications that run in an open field of interaction (Makita systems (e.g. (Pouliquen-Lardy et al. 2016; Kurup and Liu et al. 2016; You and Thompson 2017), the system must be 2016; Gauglitz et al. 2014)).The applications of computa- the aware of its own operations. This awareness is partic- tional intelligence have involved interactive elements, such ularly useful for information security and privacy manage- as virtual objects, virtual roles and real user connected by the ment. due to the possibility of public access in an open network. The issue of optimization of traffic flow was also architecture. Since information of users’ behavior and re- addressed in those works. However, iHCI systems mostly sponse will be generated during immersive interaction, this focused on the performance of the system, rather than the type information may be sensitive as well, in terms of pri- performance of user experience. vacy, especially in therapy, education, and military (El- iHCI systems and other smart systems, e.g., robotics, liott, de Bruin, and Dumoulin 2015; Merchant et al. 2014; share similar properties, such as real-time processing, power Rizzo et al. 2011). In some cases, some critical data such efficiency, computing resources, which are common issues as photo, video and body measurements may be involved in implementation. However, intelligent HCI system focus as well. When a spying node is placed in the system, the more on the performance of interaction and user experience profiles of the users would be stolen, and the system itself than a robotic system. Therefore, the user experience is an would be attacked with malware (e.g. man in the middle at- integral part of iHCI. tack malware). If the system had self-awareness to detect As for the user experience, there are many studies con- such anomalousness, it can detect the unusual changes and ducted to explore the affects on users during the interac- protect the system as well as users’ privacy. Therefore, it tion (e.g. (Loup-Escande et al. 2017; Kuliga et al. 2015; can be seen that the awareness enables the HCI systems to Olsson et al.