Proceedings of the 2009 IEEE International Conference on Systems, Man, and Cybernetics San Antonio, TX, USA - October 2009 Quality of Experience for Adaptation in Augmented Reality Damien Perritaz, Christophe Salzmann, Denis Gillet School of Engineering Ecole´ Polytechnique F´ed´erale de Lausanne (EPFL) CH-1015 Lausanne, Switzerland firstname.name@epfl.ch Abstract—Augmented Reality enhances the user perception mitted to the display using a wireless communication link, the by overlaying real world information with computer-generated transmission rate might be constrained [3]. The rate constraint information. In this paper, we study the user experience in for wireless transmission is taken as illustrative example along Augmented Reality for the deployment of real-time adaptation. In the context of thin-client mobile Augmented Reality, the the paper, but other constraints such as processing limitation rate is often constrained due to limitations in the transmission or mobile client energy can be managed in a similar manner. link. End-to-end delay, frame rate, image size and head motion With rate constraint, the image size (resolution or compression) speed have been identified as important variables impacting and the frequency (frame rate) at which the images are the user experience. We propose a model to link the effect of generated must be controlled in real time. For a better user these variables with the Quality of Experience metrics. Using this model, we present an adaptation scheme that adapts, in experience, the adaptation consists of feedback mechanisms real time, the frame rate and the image size to maximize the where measurements, such as head motion speed, are used to Quality of Experience according to the context while satisfying determine the values to apply to the AR system. Intuitively, the given rate constraint. Simulation shows the efficiency of the frame rate should be high for fast head motion, while high proposed scheme which achieves a better Quality of Experience resolution seems more important for slow motion. than without adaptation. The adaptation still performs better than a solution with a fixed frame rate set to its maximum value. The proposed adaptation scheme requires a model to ad- equately control these encoding parameters. The model gives Index Terms—Augmented Reality, Quality of Experience, the relation between frame rate, image size, delay, head motion adaptation scheme, model, mobile thin-client, rate constraint. speed and the user experience. The effect of some of these variables on user experience has been studied [4]–[6] but was I. INTRODUCTION not used for adaptation. In AR, adaptation is mainly used by Augmented Reality (AR) is mainly applied to the sense of mechanisms for real-time video transmission [7], [8]. sight by displaying 3D virtual objects onto the real vision [1]. In this paper, we first identify the variables that have an In a near future, AR would eventually be applied to many impact on user experience and present an objective model common tasks, such as museum visit, home maintenance, suitable for adaptation. This model links the above variables to domotics and marketing. In order to give the illusion that the Quality of Experience (QoE) metrics for user experience. virtual objects are parts of the reality, tracking, processing Based on the model, an adaptation scheme which maximizes and display are needed. The tracking system measures user the QoE by adjusting the input variables regarding constraint position and orientation to define the viewpoint. The virtual and other variables measurements is presented. Finally, the scene is generated and an image is rendered for the particular model is identified for a specific setup and the adaptation viewpoint. This image is then displayed to the user, for scheme is illustrated in simulation. example on an optical see-through Head Mounted Display II. MODEL (HMD). The user vision is thus enriched with an aligned virtual scene. In Augmented Reality (AR) like in other Man-Machine Most of today’s AR applications aim at providing a given Interaction applications, user experience plays a key role. To user experience. The setup parameters such as frame rate, maximize the user experience, the variables impacting the image resolution or transmission delay may not be modified subjective user experience should be known and understood. over time even if a feasible compromise would improve the The variables have been classified in three main groups, user experience. In this paper, we propose to adapt, in real namely the design variables, the encoding parameters and time, the variables that can be controlled. The user experience the contextual variables. The design variables correspond to is therefore improved by adjusting some variables according mechanical and ergonomic aspects. The screen dimension, the to the context. weight of the system, the communication infrastructure choice In mobile AR the virtual scene can be generated and or the static tracking error are included in this group. They are rendered at the server side and then sent as a video stream determined at the design stage and are generally fixed. The to the mobile client for display [2]. If the images are trans- encoding parameters are related to the realism of virtual scene 978-1-4244-2794-9/09/$25.00 c 2009 IEEE SMC 2009 904 978-1-4244-2794-9/09/$25.00 ©2009 IEEE generation, or its representation as image. It could correspond the misalignment between real and virtual worlds grows. This to the number of polygons of a virtual objects, the color depth, additional delay is barely noticeable at high frame rate but is the image resolution or the number of images rendered per annoying at lower frame rate. second. Contextual variables have also an important impact The adaptation scheme proposed in Sec. III requires a model on user experience. In a stress situation, for example when to represent the impact of the selected variables on the user a security alarm occurs, the user will not appreciate the AR experience. Experience evaluation can be either subjective or application the same way as during his usual work. AR is also objective [10]. Subjective evaluation is a reliable but time appreciated differently depending on previous knowledge or consuming quality measurement method. Subjective testing is expertise level. Other contextual variables such as user health, generally done by asking users to vote for the experiences visual acuity, the task difficulty or the moving speed also perceived with different operating conditions. However, sub- impact user appreciations. Some of these variables can be jective evaluation cannot provide real-time quality monitoring. quantified and their impact on user experience too. Objective evaluation is not as accurate as subjective evaluation, Another important variable impacting the user experience is but can be used in real time without intruding the end user. the end-to-end delay. The end-to-end delay corresponds to the For video quality assessment, the Peak Signal to Noise Ratio time difference between the instant when the virtual objects measure has been widely used. However it does not correlate should be displayed and the instant when the AR image is well subjective results. Other objective measures are more displayed. If this delay is zero, virtual and real worlds are suitable to replace subjective measures [11] for video quality perfectly aligned; if it is not, virtual objects might seem to assessment, but there is not such a measure for AR. swim around and lag behind their supposed position in the real We define the Quality of Experience as the objective metrics world. The higher the delay, the lower the coherence between of the user experience for a specific application. This metrics virtual and real worlds is. The delay can be classified either as isbasedonamodel which links the impact of the selected a design variable if it is fixed over time, or as a contextual variables on the Quality of Experience (QoE). The model is variable if it is subject to variation. In this paper, since identified with subjective user experience evaluation. information is transmitted over a wireless link, delay varies and is therefore taken as contextual variable. Since they are static, B. Mathematical representation design variables that do not change over time are not taken into The adaptation scheme requires a model represented in account in this study for real-time adaptation. Only encoding an adequate mathematical form. An hypersuface is chosen to parameters and contextual variables are considered since their link the selected variables to the QoE. An hypersurface is an evolution modifies instantaneously the user experience. higher-dimensional generalization of the surface concept. It consists of a n-dimensional topological manifold represented A. Model variables for mobile Augmented Reality in a n+1 Euclidian space, where n corresponds to the number In the case of data transmission from a server to a mobile of variables. With 4 variables, this is a 4D hypersurface in a client through a wireless link, user experience is affected by 5D space, which is not graphically displayable in the usual the video quality and by the dynamic registration error [9] 3D space. B-splines have been arbitrarily chosen to define the (static registration error is assumed to be negligible). The video model in a smooth way and with a low error sensitivity. quality depends on the encoding parameters, namely the image size and the frame rate. The dynamic registration error depends C. Model identification mainly on the end-to-end delay. The head motion speed can be The model is fully defined when the hypersurface function taken as another contextual variable that has a coupled impact coefficients are known. These coefficients are identified using with frame rate and delay on the user experience. Therefore, subjective testing with a finite set of variables combinations we consider the image size, the frame rate, the end-to-end covering the range of all possible variables values.