INTERNATIONAL ORGANISATION FOR STANDARDISATION ORGANISATION INTERNATIONALE DE NORMALISATION ISO/IEC JTC 1/SC 29/WG 11 CODING OF MOVING PICTURES AND AUDIO

ISO/IEC JTC 1/SC 29/WG 11m m23798 February 2012, San Jose, CA

Title: Additional Use Cases for ARAF Authors: Marius Preda, Ivica Arsov (on behalf of the TOTEM project)

1 Mobile Mixed / Games use case

Augmented Reality and, in general, combines the real world with a virtual scene and creates an immersive and interactive real-time environment. The possible applications are almost limitless and encompass areas like learning, architecture or industrial production. In the gaming area, AR/MR offers unique opportunities. Unlike with traditional video games, these games are not imprisoned onto the screen but interact with the real world. They incorporate real locations and objects into the game, therefore tapping into already existing thoughts and emotions of the player, which in turn creates a potentially much richer gaming world and experience. Mobile AR/MR Gaming even goes a step further as the playing area becomes borderless and games can be played literally anywhere and anytime. With the current advancement of modern cell phone technology, we are identifying the potential of Mobile AR/MR Games being played by anyone.

2 Background and state of the art

Digital games employ digital technology as part of the game experience. In contrast to non computer games, digital games “facilitate the play of games that would be too complicated in a non computerized context.”[Salen, 2003]. Game developers can create rich, interactive virtual worlds that real people can inhabit. Through digital technology and the possibility of representing complex behavior, computer games can deliver new and intense experiences to players. Mobile or pervasive games reach beyond desktop- or console- based gaming. They overcome spatial, temporal, and social boundaries of conventional computer games by making the real environment intrinsic.[Montola, 2005] They bridge the gap between traditional non computer games and existing video games by offering direct social interaction [Magerkurth, 2005]. Thus, mobile games integrate social components of traditional games into digital games, extending them into our everyday activities. Location-based or -aware games emphasize these aspects of pervasive games [Bell, 2006]. The game content and the player’s interaction opportunities depend on the player’s current real-world location. The user’s local environment (including buildings, plants, people, and so on) might contain items relevant for game play. Moreover, interaction with some of these items might be part of the game. This interaction extends to other players who might compete against one another or cooperate as part of the game when sharing a certain area—similar to common real-world activities. Finally, pervasive or mobile mixed reality (MR) games [Broll, 2006] represent distinctive location-aware pervasive games—that is, they use MR technology to enhance or modify the players’ real environment with virtual content. Instead of being represented by an avatar like in many traditional computer games, players are their own avatar moving in the game world by moving in their real environment. Most pervasive MR games are purely outdoor. However, some of them facilitate combined indoor and outdoor activities. Pervasive MR games typically combine real and virtual content, storytelling, and the user’s imagination to create a new type of entertainment experience.

Such systems have the potential to create an ultimate gaming experience [Stapleton, 2003] One of the first mobile MR (or Augmented Reality) games was ARQuake, developed in 2002 [Piekarski, 2002]. ARQuake is the AR version of the famous first-person shooter Quake. The aim is identical to the desktop version where the player shoots monsters and collects items. In the AR version, the players wear AR systems consisting of a laptop (in a backpack) attached to several sensors for tracking position such as differential GPS for the orientation and a camera for marker-based computer vision tracking. In 2004, another AR variant of a famous computer game was developed. The story of Human Pacman [Cheok, 2004] is the same as that of the original arcade game Pacman: Pacman (represented by one player), has to collect all the cookies in the world before he is caught by a ghost (represented by another player). The game’s system consists of a central server, the wearable client system, a Bluetooth device, and a helper laptop. The client and server communicate via WiFi, and the client is connected to the physical devices via Bluetooth. The user’s viewing orientation is tracked by a 3-degrees-of-freedom orientation sensor built into the helmet and the position is determined via GPS. A special sensor detects whether the player is touched by the enemy player. NetAttack [Broll, 2006] also developed in 2004, is a scavenger hunt game. The basic idea and technique of ARQuake was enhanced by having outdoor as well as indoor game play. The outdoor player wears a mobile AR system and is tracked by GPS and marker-based computer vision tracking. The outdoor player is supported by an indoor player. The partner inside, sitting at a desktop-based survey system, communicates via Wi-Fi with the outdoor player, guides him, and helps him find game items. In 2004, AR Soccer [Reimann, 2004] was the first handheld-based AR game. AR Soccer is a computer vision-based smart phone game. Players shoot a penalty by kicking a virtual ball into a virtual goal with their real feet. The smart phone’s camera tracks the player’s foot. Instead of an optical flow algorithm, the foot is detected and tracked using a 2D edge extraction approach. In 2005, AR Tennis [Henrysson, 2005] and The Invisible Train [Wagner, 2005] were developed as marker-based games on handheld devices. AR Tennis is a multiuser-supported smart phone game that also uses computer vision-based tracking. In AR Tennis, the smart phones are equipped with markers, which can be detected by the cameras of the other players’ phones. The tracking data is transmitted via a peer-to-peer Bluetooth connection, thus enabling the two players to play tennis virtually with their smart phones. The aim of the multiuser game The Invisible Train is to steer a train over a wooden railroad track. The player can interact over the touch screen by changing the speed of the trains and the switches. The Invisible Train is a synchronized multiuser game in which PDAs are connected via Wi- Fi. The tracking is realized by a marker-based computer vision approach. In Capture the Flag [Cheok, 2006] developed in 2006, a team catches the opponent’s flag, which is presented by a small wooden box. This box is equipped with a Bluetooth device and a touch sensor. The touch sensor tracks whether the participants pick or move the box. The box and the smart phone are connected via Bluetooth. Mr. Virtuoso is an educational game played as video-see-through AR on a PDA [Wagner, 2006]. The objective is to sort artwork according to their date of creation along a timeline. The timeline consists of fiducial markers. Each marker carries one artwork. The virtual expert Mr. Virtuoso provides information to the artworks. The PDAs are synchronized via WiFi using a server. In 2007, Epidemic Menace [Lindt, 2007] the successor of NetAttack, was developed. Epidemic Menace uses multiple devices including PDAs and smart phones. It also takes place indoors and outdoors, and the players communicate via smartphones. The smart phones display a small map of the game area and the user’s current position. The PDA supports position tracking. The outdoor players are optionally equipped with mobile AR systems. TimeWarp [Herbst 2008] is a Mobile Mixed Reality game set in the oldtown of Cologne. Players travel in time and space to meet and help the legendary Heinzelmännchen elves while they have to interact with virtual and real objects alike. The Alchemists [Wetzel 2008] takes a simpler approach where the players have to collect magical ingredients from the environment to brew potent potions (see Figure 2). What makes this game unique is the fact that it was built with the so-called “Magic Lens Box”, an attempt at providing an authoring environment for Augmented Reality games. The input features exploited in the games presented above are the location/orientation (thanks to a GPS system, or NFC tags/QR-codes spread all over the field…), the accelerometer, the Bluetooth/GSM/WiFi connection, and the image/video captured by the mobile phone's camera. One objective of the project is to build (based on existent APIs) a unified framework to manage and process these inputs.

A second category of information ready to be exploited in mobile mixed reality games is the user/player context. It is used to trigger the right service/action/reaction in the right moment and is at the heart of two very active research subjects named Pervasive Media and Autonomic Computing. Context is derived from sensors or from digital resources (presence services such as CellID, user input, …) and may refer to physical, virtual or social circumstances. Existent games such as ContextKing [Koolwaaij 2009] already exploit context management frameworks [Böhm 2008] for collection/aggregation/distribution of context information. For the localization data, as the main context feature, several solutions were implemented to ensure its computation whichever form of sensor used [Dey 2001, Roman 2002, Capra 2003, DaRocha 2006, Julien 2006]. To improve the accuracy or to provide higher level interpretation, sensor fusion is also an active domain reported in the literature [Fox 2001]. Some commercial efforts, while basic today and mainly based on self-reporting, are available to collect low- (e.g location) or high-level (e.g. current action) contextual data. One of the most popular is Twitter, a web service based on the question "What are you doing?" However, the high dimensionality and the very heterogeneous nature of context information are still posing technical challenges that remains to be addressed: provisioning, discovery, routing, privacy/trust/protection, diversity [Stenton 2008].

Probably the highest barrier in the development of mobile mixed reality games will remain the authoring of the content. Mainly there are today two class of approaches: the rapid prototyping toolkits providing re-usable components-widgets [10 day] or end-user programming tools, more or less intuitive to use [Hull 2004, Barrenho 2006, Biswas 2006]. In both cases, the authoring is performed at the desktop and not in real gaming environment. Other examples for MR authoring tools include MARS [Gueven 2003], AMIRE [Doerner 2002] or DART [MacIntyre 2003] all of which are not gaming specific.

Concerning user-created content typical Web2.0 applications like blogs, wikis, and video and photo community sites are already a big step in this direction. The latest developments, in the form of easily manageable and configurable web services such as mashups, go beyond the pure management of non-computational media towards creating your own programs. In the area of video games, user-created games or user-modified ones have become rather popular. Apart from the modding community of first person shooters [Kücklich, 2005] this trend has also moved over to other gaming genres as seen in games like World of Warcraft, The Sims or Spore. One of the main aspects of the latter is the encouragement of user-created content (e.g. creatures) for this game which goes so far that people pay to use the creature editor and enhance the game world with their creations.

Pervasive games like Mobile Mixed Reality games usually require a huge amount of effort not only to produce the game but also to stage it. If part of this job could be done by the community, the popularity and economical feasibility of such endeavors would most likely increase. One way is to make the set-up of the game part of the actual game play, as it is done for example done in the game Hitchers [Drozd, 2006] where players create game content themselves and place it in the environment. Full games of course are not possible to create with such an approach.

Other examples from the non-video gaming area which have managed to build a large community around them include Geocaching (http://www.geocaching.com), where users create quizzes for others based on GPS technology and Lego Mindstorms (http://mindstorms.lego.com), where users can script and define the behavior of Lego robots.

3 Technologies for Mobile MR/AR Games

Technical challenges in several domains related to Mobile MR/AR games exist: interaction mechanisms, context framework and games authoring by community. In addition, several technologies have to be deployed to ensure content (images/videos/graphics/scripts/metadata) management in the chain (capture, compression, transmission, storage, adaptation and presentation) and multi-user functionalities (game sessions, lobby servers, IDs). In order to create compelling Mobile MR/AR Games, tracking plays an important role as the actual position of the player in the real world translates to his virtual position: The player becomes his own avatar. There are already several tracking mechanisms well established ranging from GPS tracking, inertial sensor tracking, visual marker- based tracking to more complex like feature based tracking. Deriving high level context information from relatively low level sensor data, ready to be exploited in applications such as games, is still an open issue for mobile multiplayer games. High-level context abstraction is needed, as well as methods of interpreting and translating sensor data to infer meaningful context situations. Such abstraction level allows as well classifying the information and uses only the one most appropriate for a specific situation of the game play. Compact representation, i.e. coding of this contextual information is also necessary, since it is expected to transmit it to the other players. In general, the games are one of the most complex multimedia applications. They represent a complete ecosystem, being composed of media assets (images/graphics/videos) but also a programming layer implementing the behaviour during the game-play and tools to capture and interpret different kind of inputs. Authoring games is thus a complex task, involving different participants and to keep the development consistent, a certain level of orchestration is necessary. Opening this task to the community requires making available powerful yet simple tools and mechanisms. Since the main target of the project is to let users/players to create games, including the multimedia assets, interoperability is an issue. Moreover, the data to be transmitted should be as compact/compressed as possible. MPEG-4 already provides a set of tools for representing audio/video and graphics assets. Context data as well as relevant sensory data could be digitized and encapsulated as well in MPEG-4/MPEG-V streams. Such integrated approach ensures multimedia delivery synchronization and use of standard transport protocols. Other than on the level of media assets, interoperability should be ensured as well for the game behaviour, scene description, input mechanisms and communication protocols. Creation and playing processes will involve several collaborating users. To take care of their data communication, existent middleware can be used and integrate them with the MPEG Player.

4 Acknowledgement The Mobile Mixed Reality/Augmented Reality use case was prepared together with Fraunfoher FIT as part of the TOTEM project (www.totem-games.org), financially supported by German Federal Ministry of Education and Research and by the French National Research Agency. 5 References

Ahn, L. von, Dabbish, L., Labeling images with a computer game, Conference on Human Factors in Computing Systems archive, Proceedings of the SIGCHI conference on Human factors in computing systems table of contents, Vienna, Austria, pp 319 – 326, 2004, ISBN:1-58113-702-8 Barrenho, F., et al. (2006). InAuthoring environment: interfaces for creating spatial stories and gaming activities. Proc. 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, ACM. Bell, W. et al., “Interweaving Mobile Games with Everyday Life,” Proc. SIGCHI Conf. Human Factors in Computing Systems (CHI 06), ACM Press, 2006, pp. 417–426. Biswas, A., et al. (2006). Assessment of mobile experience engine, the development toolkit for context aware mobile applications. Proc. 2006 ACM SIGCHI international conference on Advances in computer entertainment technology, ACM. Böhm, S., Koolwaaij, J., and Luther, M., Share whatever you like , In Proceedings of the First International DisCoTeC Workshop on Context-aware Adaptation Mechanisms for Pervasive and Ubiquitous Services (CAMPUS’08), June 2008 Broll, W., Lindt, I., Ohlenburg, J., Herbst, I., Braun, A.-K., and Wetzel, R.: „Toward Next-Gen AR Games”, IEEE Compute r Graphics & Applications, Vol. 11, No. 4, 2008, Special Issue on Mobile Graphics, IEEE Computer Society, Piscataway, NJ, USA, 24 - 32. Broll, W. et al., “Meeting Technology Challenges of Pervasive Augmented Reality Games,” Proc. ACM NetGames 06, ACM Press, 2006, article no. 28; http://doi.acm.org/10.1145/1230040.1230097. Broll, W., Herling, J., and Blum, L.: „Interactive Bits: Prototyping of Mixed Reality Applications and Interaction Techniques through Visual Programming”, in Proc. of the 3rd IEEE International Symposium on 3D User Interfaces 2008 (IEEE 3DUI 2008), Reno, Nev., USA, March 8-9, 2008, IEEE Computer Society, Piscataway, NJ, USA, 109- 115. Capra, L., Emmerich, W., and Mascolo, C. CARISMA: Context-Aware Reflective Middleware System for Mobile Applications. IEEE Transactions on Software Engineering 29, 10 (Oct. 2003), 929–945. Cheok et al., “Capture the Flag: Mixed-Reality Social Gaming with Smart Phones,” IEEE Pervasive Computing, vol. 5, no. 2, 2006, pp. 62–69. Cheok et al., “Human Pacman: A Mobile, Wide-Area Entertainment System Based on Physical, Social, and ,” Personal Ubiquitous Comput., vol. 8, no. 2, 2004, pp. 71–81. Conan, D., Rouvoy, R. and Seinturier, L. Scalable Processing of Context Information with COSMOS. Proc. 7th IFIP International Conference on Distributed Applications and Interoperable Systems, Springer-Verlag, Lecture Notes in Computer Science Volume 4531, pages 210-224, Paphos, Cyprus, June 2007 Couderc P. and Banâtre M., Beyond RFID: The Ubiquitous Near-Field Distributed Memory, ERCIM News, no 76, January 2009, pp 35-36. da Rocha, R., and Endler, M. Context Management in Heterogeneous, Evolving Ubiquitous Envrionments. IEEE Distributed Systems Online 7, 4 (Apr. 2006), 1–13. Dey, A. K., D., Abowd, G. D., & Salber, D. (2001). A conceptual framework and a toolkit for supporting the rapid prototyping of context-aware applications. Human-Computer Interaction, 16 (2-4):97-166. Doerner, R., Geiger, C., Haller, M., Paelke, V. Authoring Mixed Reality. A Component and Framework-Based Approach. In First International Workshop on Entertainment Computing (IWEC 2002), May 2002. Drozd, A., et al.: Hitchers: Designing for Cellular Positioning. In proceeding of UbiComp 2006: Ubiquitous Computing. Lecture Notes in Computer Science Volume 4206/2006, pp. 279-296. Fox, D., Hightower, J., Liao, L., Schulz, D., and Borriello, G. (2003). Bayesian filtering for location estimation. IEEE Pervasive Computing, 2(3): 24-33. Gueven, S., Feiner, S.. Authoring 3D Hypermedia for Wearable Augmented and . In Proceedings of ISWC’03, pages 118–126, 2003. Henrysson, M. Billinghurst, and M. Ollila, “Face to Face Collaborative AR on Mobile Phones,” Proc. Int’l Symposium Mixed and Augmented Reality (ISMAR 2005), IEEE CS Press, 2005, pp. 80–89. Herbst, I., Braun, A., McCall, R., and Broll, W. 2008. TimeWarp: interactive time travel with a mobile mixed reality game. In Proceedings of the 10th international Conference on Human Computer interaction with Mobile Devices and Services (Amsterdam, The Netherlands, September 02 - 05, 2008). MobileHCI '08. ACM, New York, NY, 235- 244. Hu, P., Indulska, J. and Robinson, R. An Autonomic Context Management System to Pervasive Computing. In Proc. 6th IEEE International Conference on Pervasive Computing and Communications, pages 213–223, Hong Kong, March 2008. Hull, R., Clayton, B., and Melamed, T. (2004). Rapid Authoring of Mediascapes. Proc. UbiComp 2004. Julien, C., and Gruio-Catalin, R. EgoSpaces: Facilitating Rapid Development of Context-Aware Mobile Applications. IEEE Transactions on Software Engineering 32, 5 (May 2006), 281–298. Kephart, J.O. and Chess, D.M. The Vision of Autonomic Computing. IEEE Computer, 36(1), January 2003. Koolwaaij, J., Wibbels, M., Böhm, S., and Luther, M., ContextKing: Virtual Kingdoms in Real Life, In Proceedings of the first IEEE International Conference in Games and Virtual Worlds for Serious Applications (IEEE VS- Games'09), March 23 - 24, 2009, Coventry, UK, March 2009. Kücklich, Julian: Precarious Playbour: Modders and the Digital Games Industry. In Fibreculture Journal, ISSN: 1449 – 1443 Issue 5(2005), Australia Lindt et al., “A Report on the Crossmedia Game Epidemic Menace,” ACM Computers in Entertainment (CIE), vol. 5, no. 1, 2007, article no. 8; http://doi.acm. org/10.1145/1236224.1236237 Lindt, I., Broll, W.: “NetAttack – First Steps Towards Pervasive Gaming”, ERCIM NEWS Special Issue on Games Technology, No. 57 (April 2004,). 49-50 MacIntyre, B., Gandy, M., Bolter, J., Dow, S. and Hannigan, B. "DART: The Designer’s Augmented Reality Toolkit", In Proc. of IEEE and ACM International Symposium on Mixed and Augmented Reality, Tokyo, Japan, 2003, pp. 329-330. Magerkurth et al., “Pervasive Games: Bringing Computer Entertainment Back to the Real World,” Comput. Entertain., vol. 3, no. 3, 2005, p. 4. McCarthy, C., International flavor comes to OpenSocial with translation app, CNET News, September 24, 2008, available online at http://news.cnet.com/8301-17939_109-10050621-2.html Montola, M., “Exploring the Edge of the Magic Circle. Defining Pervasive Games,” Proc. Proc. Of Digital Experience: Design, Aesthetics, Practice Conf., 2005; http://users.tkk.fi/~mmontola/exploringtheedge.pdf. Paspallis, N., Rouvoy, R., Barone, P., Papadopoulos, G.A., Eliassen, F. and Mamelli, A. A Pluggable and Reconfigurable Architecture for a Context-aware Enabling Middleware System. In Proc. 10th International Symposium on Distributed Objects and Applications, volume 5331 of Lecture Notes in Computer Science, pages 553–570, Monterrey, Mexico, November 2008. Springer-Verlag. Piekarski, W. and Thomas, B., “ARQuake: The Outdoor Augmented Reality Gaming System,” Commun. ACM, vol. 45, no. 1, 2002, pp. 36–38. Reimann, C., Paelke, V., and Stichling, D., “Foot-Based Mobile Interaction with Games,” Proc. ACM SIGCHI Int’l Conf. Advances Computer Entertainment Technology (ACE 04), vol. 74, ACM Press, 2004, pp. 321–324. Román, M., Hess, C., Cerqueira, R., Ranganathan, A., Campbell, R., and Nahrstedt, K. Gaia: A Middleware Infrastructure to Enable Active Spaces. IEEE Pervasive Computing 1, 4 (Oct. 2002), 74–83. Rouvoy, R., Conan, D. and Seinturier, L. Software Architecture Patterns for a Context Processing Middleware Framework IEEE Distributed Systems Online, Volume 9, Number 6, June 2008. Salen, K. and E. Zimmermann, E., Rules of Play. Game Design Fundamentals, MIT Press, 2003. Stapleton, C.B. and Hughes, C.E., “Interactive Imagination: Tapping the Emotions through Interactive Story for Compelling Simulations,” IEEE Computer Graphics and Applications, vol. 24, no. 5, 2003, pp. 11–15. Stenton, Phil; Hull, Richard; Morgan, Jeff; Kindberg, Tim; Pollard, Stephen; Hunter, Andrew; Reid, Josephine; Hawkes, Rych; Wainwright, Nick Pervasive Media: Delivering 'the right thing in the moment', HP Laboratories, HPL-2008-124, Submitted to 2008 NEM submit, St Malo, France, Oct 13, 2008, available on-line at http://www.hpl.hp.com/techreports/2008/HPL-2008-124.html Sterritt R., Parashar M., Tianfield, H. and Unland, R. A concise introduction to autonomic computing. Advanced Engineering Informatics, 19(3):181–187, July 2005. Tran S.M., Preda M., Prêteux F., Fazekas K. Exploring MPEG-4 BIFS features for creating multimedia games Proceedings IEEE International Conference on Multimedia and Expo(ICME'03), Baltimore, WA, Vol. 1, 2003, p. 429-432. Wagner, D. et al., “Towards Massively Multi-User Augmented Reality on Handheld Devices,” Proc. 3rd Int’l Conf. Pervasive Computing, Springer, Berlin/Heidelberg, 2005, pp. 208–219. Wagner, D., Schmalstieg, D., and Billinghurst, M., “Handheld AR for Collaborative Edutainment,” Proc. 16th Int’l Conf. Artificial Reality and Telexistence (ICAT), Springer, Berlin/Heidelberg, 2006, pp. 85–96. Wetzel, R., Lindt, I., Waern, A., and Johnson, S. 2008. The magic lens box: simplifying the development of mixed reality games. In Proceedings of the 3rd international Conference on Digital interactive Media in Entertainment and Arts (Athens, Greece, September 10 - 12, 2008). DIMEA '08, vol. 349. ACM, New York, NY, 479-486. Wittkaemper, M., Broll, W., Lindt, I. Proceedings of MRUI 2006 - First Workshop on Mixed Reality User Interfaces - Approaches, Languages, Standardization" at IEEE Virtual Reality 2006, pp. 20-24, also in DVD Proc. of IEEE VR 2006, Alexandria, Virginia, USA, 2006.