Hacking Nintendo Wii to paint virtual graffiti A low cost virtual tangible tool for users involvement Paola Salomoni Ludovico Antonio Muratori Department of Computer Science Department of Computer Science University of Bologna University of Bologna Bologna, Italy Bologna, Italy [email protected] [email protected] Silvia Mirri Francesco Pozzi Department of Computer Science Corso di Laurea in Scienze e Tecnologie Informatiche University of Bologna University of Bologna Bologna, Italy Bologna, Italy [email protected] [email protected] Abstract —Any approach to the issues of virtual reality implies the market. In particular, all the power of interfaces, which a general question about how to effectively plunge inside it. have been designed to involve players as much as possible, Despite a great variety of settings, the more feedbacks are the can be exploited on PC platform because of they are same as real ones, the more users feel involved on a likely compliant with well-known and documented standards. situation. We can split such an assumption into the necessity of Bug fixing and enhancing (also cracking) which were effective gestural recognition and likelihood of virtual tools to done with the early, so-called mods by a small group of reality. In this paper we will present a system to draw virtual electronics insiders, have given way to hundreds of graffiti on the PC screen, as it were a wall. A virtual, but homebrew works (i.e. hackings of the consoles systems) by a tangible aerosol bomb has been designed and built, by large community of fans. Besides the producers’ exploiting the Nintendo console gaming device and sensor documentation about consoles internals, standard system. Our object has the same shape and the same way of use as a common, real spray paint dispenser and it can be used on specifications and official (or unofficial) SDK’s allows to quite every available graphics application on a Desktop or explore and re-invent uses and capabilities of gaming notebook case. Besides the advantages due to the low costs and devices. Input and feedback consoles systems, together with the good accuracy of gesture recognition, our system confirms standard compliant characteristics let designing, building and the importance of similarity in simulated scenarios. assessing a wide range of low costs applications linked to gesture recognition. As for technological and scientific Pervasive computing, gestural interface, multimodal research, innovative features of the Nintendo console game interface, virtual tool. pad – the Wiimote – and its input system, have been exploited to build instruments for rehabilitation [2,3], I. INTRODUCTION augmented reality [4], and, in general, for gestural interfaces. In this paper we will present a system and a gestural Users’ immersions on virtual environments are tightly interface to draw virtual graffiti on a personal computer connected to the similarity of simulated objects they use and monitor or connected projector. It is made up of a virtual gestures they can do, as regards to the real ones. High costs aerosol bomb, which has been built by utilizing the Wiimote of sensors, transducers and computer-vision systems which device and its sensors system. Our goal has been can effectively detect and classify movements and actions, investigating the potential of such a low cost equipment to have somehow limited advance and trials on this field. The realistically simulate human activities and assessing the recent encroachment of gaming consoles, as pointed out in limits of driving such a device as a PC peripheral. Unlike the [1], on PC realm seems to make such costs more affordable original architecture of the Nintendo console, which uses the and opens new perspectives on approaching issues and remote controller to receive red light signals from a sensor implementations for immersion on virtual realities. The same bar source and to transmits some kinetics parameters to the standards which are usual on a desktop case or a notebook, system via Bluetooth, our approach inverts the flow of such as hard drives, graphics cards and communication information. Instead of using the remote controller as a devices are now embedded on the foremost game stations of 9781-4244-3941-6/09/$25.00 ©2009 IEEE receiver, we chose to exploit it as a sensor system, at http://news.cnet.com/8301-17938_105-10253586-1.html recognizing red light emitters from the virtual aerosol bomb. for details). Interests toward such systems are witnessed by Common graphics applications, such as The Gimp, some companies such as, as an example, Softkinetics, which Photoshop or MSPaint, can be used as virtual environments promises the development of a “standard 3D real-time (the wall, the spray paint store, etc.) to draw. An approach gesture recognition platform for Interactive digital oriented to device driving, which does not depend on the entertainment, by means of a middleware which insulates applications has been adopted so as to guarantee the interface game developers from the low-level technicalities” (at: portability. Since it is completely hand-in-glove as regard to http://www.softkinetic.net/Public/). the real one, the virtual aerosol bomb warrants a full Virtual tools, meant as object which look (in shape and likelihood with real gestures and proved the importance of way of use) like the real ones, can be considered as a must on the above hinted similarity for full immersion on virtual enthralling simulations. Furthermore, they can significantly reality. reduce the computation to recognize any gesture or action. The remainder of this paper is organized as follows: As shown by Fuentes at al. on [18], general customization of section II summarizes the background and related work the interface freedom-degrees, up to obtain a specific “virtual about gestural interfaces and hacking approaches to the tool”, implies a lower amount of evaluations and processing consoles world, section III detail the design issues of our to make it effectively work. Catalogues of gaming consoles system, by pointing out hardware and software features. accessories are filled up with virtual tools: from the guns for Section IV reports some conclusions and discloses some simulated shot, to the car or plane console-kits with cloche, future work we planned to do. steering wheel, pedals and so on. In particular, the Nintendo Wii console has proposed an innovative system utilizing a CV system (extremely II. BACKGROUND AND RELATED WORK simplified) and a quite universally adaptable device – the Dan Saffer’s book “Designing Gestural Interfaces: Wiimote - capable to be transformed into a specialized tool Touchscreens and Interactive Devices” [5] has been defined with few custom dap joint parts (at “a testament to the fact that tangible, interactive interfaces http://www.nintendo.com/wii/what). The Bluetooth standard that move beyond the mouse and keyboard have become an communication protocol is used to connect the Wiimote and important part of product design”. [6] infrared light emitters are recognized and evaluated to Everything in utilizing gestures to computer applications provide kinetics parameters (position, speed, acceleration) to began with the PhD work of Ivan Sutherland [7], where he a CV system. Simplicity and standardization of Nintendo showed an early form of stroke-based gestures using a light Wii (and Wiimote) provided a wide community of fans to pen to manipulate graphical objects on a tablet display. Since design and build their own application of it as the so called then, stroke-based gesture interactions has become more and homebrew hacs (at http://wiibrew.org/ for details). On the more commonly used for text input on personal digital research field, Wiimote has been used for computer assistants (PDAs), mobile computing, and pen-based devices simulated arm and wrist therapy in stroke survivors in [2], [8, 9]. Gloves augmented with electronic motion and low cost robotics for rehabilitation on [3] and other works position sensors were developed to enhance interactions with exploiting accelerometers and gyroscopes. virtual reality applications [10, 11, 12]. A first (computer) From the point of view of the CV system, Chung Lee has vision-based system that enabled gestures to control the proposed some applications to use red light emitters for 3D volume and channel functions of a television was presented space exploration, augmented reality interaction and object on 1995 on [13]. Such a work presents the novelty of recognition on [4]. Virtual pens, head tracking glasses, and exploiting perceptual, device-free gestures, but it remains a other “virtual tools” are shown as examples to effectively technique restricted to laboratory studies [14]. employ the Wii sensor (or CV) system. Computer Vision (CV for the sake of brevity, from here), Lee’s approach points out the plenty of available as a means to allow more natural gesture interactions with software and documentation provided by the Wii homebrew computers, has been an unexploited realm till a short time community and the extremely low cost of equipments to be ago in comparison with direct-input devices such as mice used. His basic, killer idea to reverse the system data flow, [15], stylus [9, 16], or electronic gloves [17]. Such a shortage which the console was set for, has been taken into account or delay is maybe due to the fact that CV wasn’t able to here to design and build our virtual aerosol bomb. effectively satisfy accuracy constraints [14] and because of its high costs. Consumer electronics for entertainment has reduced such costs with its last generation of gaming III. DESIGN ISSUES consoles. Together with more and more power (in graphics, The availability of software tools and documentation processing, interfacing with players) nowadays majors’ about internals has made possible to design and build our trends propose equipments more and more compliant to the tool, by hacking the Nintendo console remote controller. In PC standards [1] and provide CV systems to get players the following, we will detail the scenario and all the phases more and more involved on games.