Hacking Nintendo Wii to paint virtual graffiti A low cost virtual tangible tool for users involvement

Paola Salomoni Ludovico Antonio Muratori Department of Computer Science Department of Computer Science University of Bologna University of Bologna Bologna, Italy Bologna, Italy [email protected] [email protected]

Silvia Mirri Francesco Pozzi Department of Computer Science Corso di Laurea in Scienze e Tecnologie Informatiche University of Bologna University of Bologna Bologna, Italy Bologna, Italy [email protected] [email protected]

Abstract —Any approach to the issues of virtual reality implies the market. In particular, all the power of interfaces, which a general question about how to effectively plunge inside it. have been designed to involve players as much as possible, Despite a great variety of settings, the more feedbacks are the can be exploited on PC platform because of they are same as real ones, the more users feel involved on a likely compliant with well-known and documented standards. situation. We can split such an assumption into the necessity of Bug fixing and enhancing (also cracking) which were effective gestural recognition and likelihood of virtual tools to done with the early, so-called mods by a small group of reality. In this paper we will present a system to draw virtual electronics insiders, have given way to hundreds of graffiti on the PC screen, as it were a wall. A virtual, but homebrew works (i.e. hackings of the consoles systems) by a tangible aerosol bomb has been designed and built, by large community of fans. Besides the producers’ exploiting the Nintendo console gaming device and sensor documentation about consoles internals, standard system. Our object has the same shape and the same way of use as a common, real spray paint dispenser and it can be used on specifications and official (or unofficial) SDK’s allows to quite every available graphics application on a Desktop or explore and re-invent uses and capabilities of gaming notebook case. Besides the advantages due to the low costs and devices. Input and feedback consoles systems, together with the good accuracy of gesture recognition, our system confirms standard compliant characteristics let designing, building and the importance of similarity in simulated scenarios. assessing a wide range of low costs applications linked to gesture recognition. As for technological and scientific Pervasive computing, gestural interface, multimodal research, innovative features of the Nintendo console game interface, virtual tool. pad – the Wiimote – and its input system, have been exploited to build instruments for rehabilitation [2,3], I. INTRODUCTION augmented reality [4], and, in general, for gestural interfaces. In this paper we will present a system and a gestural Users’ immersions on virtual environments are tightly interface to draw virtual graffiti on a personal computer connected to the similarity of simulated objects they use and monitor or connected projector. It is made up of a virtual gestures they can do, as regards to the real ones. High costs aerosol bomb, which has been built by utilizing the Wiimote of sensors, transducers and computer-vision systems which device and its sensors system. Our goal has been can effectively detect and classify movements and actions, investigating the potential of such a low cost equipment to have somehow limited advance and trials on this field. The realistically simulate human activities and assessing the recent encroachment of gaming consoles, as pointed out in limits of driving such a device as a PC peripheral. Unlike the [1], on PC realm seems to make such costs more affordable original architecture of the Nintendo console, which uses the and opens new perspectives on approaching issues and remote controller to receive red light signals from a sensor implementations for immersion on virtual realities. The same bar source and to transmits some kinetics parameters to the standards which are usual on a desktop case or a notebook, system via Bluetooth, our approach inverts the flow of such as hard drives, graphics cards and communication information. Instead of using the remote controller as a devices are now embedded on the foremost game stations of

9781-4244-3941-6/09/$25.00 ©2009 IEEE receiver, we chose to exploit it as a sensor system, at http://news.cnet.com/8301-17938_105-10253586-1.html recognizing red light emitters from the virtual aerosol bomb. for details). Interests toward such systems are witnessed by Common graphics applications, such as The Gimp, some companies such as, as an example, Softkinetics, which Photoshop or MSPaint, can be used as virtual environments promises the development of a “standard 3D real-time (the wall, the spray paint store, etc.) to draw. An approach gesture recognition platform for Interactive digital oriented to device driving, which does not depend on the entertainment, by means of a middleware which insulates applications has been adopted so as to guarantee the interface game developers from the low-level technicalities” (at: portability. Since it is completely hand-in-glove as regard to http://www.softkinetic.net/Public/). the real one, the virtual aerosol bomb warrants a full Virtual tools, meant as object which look (in shape and likelihood with real gestures and proved the importance of way of use) like the real ones, can be considered as a must on the above hinted similarity for full immersion on virtual enthralling simulations. Furthermore, they can significantly reality. reduce the computation to recognize any gesture or action. The remainder of this paper is organized as follows: As shown by Fuentes at al. on [18], general customization of section II summarizes the background and related work the interface freedom-degrees, up to obtain a specific “virtual about gestural interfaces and hacking approaches to the tool”, implies a lower amount of evaluations and processing consoles world, section III detail the design issues of our to make it effectively work. Catalogues of gaming consoles system, by pointing out hardware and software features. accessories are filled up with virtual tools: from the guns for Section IV reports some conclusions and discloses some simulated shot, to the car or plane console-kits with cloche, future work we planned to do. steering wheel, pedals and so on. In particular, the Nintendo Wii console has proposed an innovative system utilizing a CV system (extremely II. BACKGROUND AND RELATED WORK simplified) and a quite universally adaptable device – the Dan Saffer’s book “Designing Gestural Interfaces: Wiimote - capable to be transformed into a specialized tool Touchscreens and Interactive Devices” [5] has been defined with few custom dap joint parts (at “a testament to the fact that tangible, interactive interfaces http://www.nintendo.com/wii/what). The Bluetooth standard that move beyond the mouse and keyboard have become an communication protocol is used to connect the Wiimote and important part of product design”. [6] infrared light emitters are recognized and evaluated to Everything in utilizing gestures to computer applications provide kinetics parameters (position, speed, acceleration) to began with the PhD work of Ivan Sutherland [7], where he a CV system. Simplicity and standardization of Nintendo showed an early form of stroke-based gestures using a light Wii (and Wiimote) provided a wide community of fans to pen to manipulate graphical objects on a tablet display. Since design and build their own application of it as the so called then, stroke-based gesture interactions has become more and homebrew hacs (at http://wiibrew.org/ for details). On the more commonly used for text input on personal digital research field, Wiimote has been used for computer assistants (PDAs), mobile computing, and pen-based devices simulated arm and wrist therapy in stroke survivors in [2], [8, 9]. Gloves augmented with electronic motion and low cost robotics for rehabilitation on [3] and other works position sensors were developed to enhance interactions with exploiting accelerometers and gyroscopes. virtual reality applications [10, 11, 12]. A first (computer) From the point of view of the CV system, Chung Lee has vision-based system that enabled gestures to control the proposed some applications to use red light emitters for 3D volume and channel functions of a television was presented space exploration, augmented reality interaction and object on 1995 on [13]. Such a work presents the novelty of recognition on [4]. Virtual pens, head tracking glasses, and exploiting perceptual, device-free gestures, but it remains a other “virtual tools” are shown as examples to effectively technique restricted to laboratory studies [14]. employ the Wii sensor (or CV) system. Computer Vision (CV for the sake of brevity, from here), Lee’s approach points out the plenty of available as a means to allow more natural gesture interactions with software and documentation provided by the Wii homebrew computers, has been an unexploited realm till a short time community and the extremely low cost of equipments to be ago in comparison with direct-input devices such as mice used. His basic, killer idea to reverse the system data flow, [15], stylus [9, 16], or electronic gloves [17]. Such a shortage which the console was set for, has been taken into account or delay is maybe due to the fact that CV wasn’t able to here to design and build our virtual aerosol bomb. effectively satisfy accuracy constraints [14] and because of its high costs. Consumer electronics for entertainment has reduced such costs with its last generation of gaming III. DESIGN ISSUES consoles. Together with more and more power (in graphics, The availability of software tools and documentation processing, interfacing with players) nowadays majors’ about internals has made possible to design and build our trends propose equipments more and more compliant to the tool, by hacking the Nintendo console remote controller. In PC standards [1] and provide CV systems to get players the following, we will detail the scenario and all the phases more and more involved on games. Sony Playstation II and of our project. III, as well as Microsoft Xbox have been enriched with cameras and systems for a whole-body gaming approach, so as to “shoot the Wii” market killer product by Nintendo (see

+Z connect the remote controller to a PC, so as to recognize and map all the possible signals. In general, they exploit The -Y Pitch Bluetooth HID (Human Interface Device) specification and +X have provided some solutions to make up the partial incompatibility of Wiimote as regards to the standard. While libraries can be used inside some code to drive any specific application, device-driver programs allow to map the Wiimote input/output data for general-purpose, by setting a Yaw certain group of on-demand parameters. We chose such a +Y second paradigm to interface our virtual tool in order to -X guarantee the portability of it as a peripheral for the state of Roll the art applications, instead of writing dedicated code. -Z Specifically, we referred to the GlovePie project by Carl Kenner (see http://carl.kenner.googlepages.com/glovepie for Figure 1. Wiimote freedom-degrees details) which has been written as a customizable device- driver for the Wiimote (besides the mapping of any other device as a mice or a keyboard). Details about such software and its utilization for our goals will be disclosed in the A. Nintendo Wii at a glances following of this paper. After one year from its release, on November 2006, the 5th gaming console by Nintendo had sold 20 million pieces, C. Reversing the flow despite it was (objectively) evaluated to be underpowered in comparison with its competitors, such as Sony Playstation As for the intention of Wii designers, the CV system of and Microsoft Xbox. Nintendo console expects the Wiimote to detect infrared Nintendo’s market success is due to the innovative light emitters from the sensor bar and then to send their characteristics of its creature and, in particular, to the novelty position via Bluetooth to the console for processing, like any of its console game controller: the Wiimote. It is a wireless other parameters from buttons or from the accelerometer. handheld device communicating with the console via Moving the Wiimote provides a different point of view for microwaves through the Bluetooth protocol. Besides some its camera as regards to the emitters on the still sensor bar, so buttons, it contains a vibration engine to simulate trembling as to trace the user’ displacements. Figure 2 shows the above feedbacks, a speaker, a 3-axis accelerometer capable to trace scheme. inclination, speed and acceleration of the controller, and a The console dataflow system can be easily reversed, by high speed infrared camera. placing and driving (i.e. switching) infrared light emitters as On Figure 1 the freedom degrees (x, y and z) and the they were mobile parts of a tool and by putting the referring inclination directions (Pitch, Yaw and Roll) detected by the system in charge of the still Wiimote. Figure 3 synthesizes accelerometer are depicted. The infrared camera (from the such a reversed approach. +Y side on figure 1) is capable to map up to 4 distinct light This way, the remote controller will identify and detect sources on a 2D space at a 45-degrees angle. the displacement of really moving objects and, above all, it Some infrared led emitters are grouped on a bar (called, will be possible building devices which are essentially made with a sort of reversed term, sensor bar ) which has to be of switch and infrared lights. Our virtual aerosol bomb has placed in front of the players (usually over or upon the TV been built by exploiting the advantages of reversed data screen) and works as a referring system for the Wiimote flow. Also a system based on the direct, default approach has camera. been designed, and we will hint about it in the following.

B. Wii Homebrewing microwave As Chung Lee wrote on [4], official specifications of Moving Wiimote are unpublished, but the wide community of the so called homebrew hackers has collectively made up for the Wiimote lack of them. The most part of such an effort is available at http://wiibrew.org, a sort of de facto, open and official source console of information about Wii internals. Furthermore, the standard Bluetooth protocol used to connect the Wiimote to the console allows using it with a personal computer without infrared resorting to the development tools provided by Nintendo, which greatly limit their usage from a legal point of view. Software libraries and standalone device-driver Still Sensor Bar applications have been written and are maintained for Microsoft Windows, Apple OS X and Linux platforms, to Figure 2. The original flow of data on Nintendo Wii

Moving tool

infrared

Figure 4. The first prototype of virtual aerosol bomb. console

Still Wiimote microwave

Figure 3. The reversed approach to data flow on Nintendo Wii.

D. Hardware The Wiimote controller is able to map up to 4 distinct light-point on a 2D space. It means that the appearance and disappearance of some of them can be used to detect some kind of binary action (on/off) and the others can be in charge of tracing the object position. Furthermore, given an initial Figure 5. The can under construction (A) and the final result (B). setting, i.e. a default distance, a couple of light point emitters can be utilized to trace the actual closeness or remoteness from the still Wiimote. All this features have been taken into account to design and build our virtual aerosol bomb. A very All the functionalities of our virtual aerosol bomb are first prototype has been carried out by placing three infrared provided by processing signals coming from infrared lights. light led emitters on a box. Figure 4 shows the result: the In order to exploit the buttons placed on the Wiimote, a external blue bulbs are the light points for mapping the direct approach and a different virtual tool has been build. It device position and distance and they are switched on by the consists of a box which contains the remote controller with a button on the left, while the central one is switched by the mirror at its top. As shown on Figure 6, lights from a still button on the right so as to simulate the sprinkling. sensor bar are deflected on the camera and one of the Some experiments with this prototype have shown that Wiimote buttons is used to simulate the sprinkling. Such a single leds are not effectively detected because of their low solution has been discarded because of the resulting object is emission of light and the presence of mutual interferences. not so hand-in-glove as the above one we described. Due to the goal of producing a tool which really looks like the real one, in shape and way of use, a common spray paint dispenser has been chosen as the cover or shell for it. E. Software Three holes on the cylindrical surface of the spray can have The final goal of our project is utilizing the virtual been filled with groups of infrared light emitters. A button to aerosol bomb to draw virtual graffiti on a PC screen or a switch on the external groups of lights has been put on the projector connected to a desktop or notebook case. In bottom, while another switch has been connected to the particular, this activity will have to be possible with actuator at the top of the can. Figure 5A shows one of the widespread graphics applications such as MsPaint, construction phases: as we can see, each light-point is made Photoshop, TheGimp, which let the user choose to freehand of 9 leds instead of 1 at the external holes and of 5 leds in the sprinkling. central hole, so as to provide enough, distinguishable Based on our approach, driving our virtual tool is infrared emission. Figure 5B shows the final result. equivalent to processing the signals coming from the infrared A very simple circuit has been assembled to control the camera on the (still) Wiimote and re-transmitted to the PC lights. Figure 6 reports its schema: (b) is the switch for the; via Bluetooth. The parsing and elaborating process of such a (d) and (f) are the groups of lights; (c) is under the can data flow can be done by means of libraries linked to the actuator to switch on the (e) group of lights and, finally, (a) applications which use the Wiimote. Wiiuse by Michael is the group of batteries powering the system. Laforest, (on http://www.wiiuse.net/) is a notable instance of

such software tools, available to the wide community of Wii if (var. detect ==2) then var.CursoreX=(1024- homebrewers. Nevertheless, this paradigm of implementation implies a dedicated use of the device, strictly ((Wiimote1.Dot1x+Wiimote1.Dot2x)/2))/1024*var.screenW connected to the application which has been written for, and var.CursoreY=(var.screenH-(((Wiimote1.Dot1y so it is impossible or very complex to use it for our goals. In +Wiimote1.Dot2y)/2)/768*var.screenH)) mouse.x = var.CursoreX/var.screenW order to surmount this limit, it is necessary to place a suitable mouse.y = var.CursoreY/var.screenH set of controls at the device-driver layer as it happens for end if mouse, keyboard, and all the common PC peripherals. A satisfactory solution for this approach is GlovePie by Karl if (var.rilevati!=3) then mouse.LeftButton = false Kenner. It is a sort of meta-device driver, letting to detect end if and associate data coming from a wide variety of devices (mice, keyboards, pads and the Wiimote). In particular, Code 2. Midpoint and mouse movement association. GlovePie is a standalone application, overriding the device drivers of the operative system (only MsWindows at the time the authors write). Once launched, it captures events coming from peripherals and can process them, or associate such Some chunks of script code about all this operations are events to others. All the rules and the processing functions summarized in Code 1: the first two statements assign the are taken from a script, written in a c-like syntax, which resolution parameters (known in GlovePie) to the global users can modify or create. Global or local variable can be variables; the if…then…else statement controls the created to set up suitable parameters, as well as conditions or number of lights switched on, referring to the dot1vis and loops can be stated on the strength of received events and so dot2vis as the ones tracing the spray can position and to on. the dot3vi s as the sprinkling simulator. GlovePie scripts are to be conceived as event-driven As the Code 2 shows, the computed midpoint is programs to response any peripheral change of state. This associated to the mouse movement (first if…then software is capable to associate events coming from the statement) and the lighting of the central group of leds to the Wiimote so that they simulate other events from an already- right-click (second if…then statement). In this Code, driven peripheral like mice or keyboards. Since the latter Wiimote1.Dot1x and Wiimote1.Dot1y are the coordinates of ones are typically used to draw with the above mentioned the first group of leds on the can GlovePie is able to graphics application, such features meet our issues. automatically detect, while Wiimote1.Dot2x and For our purposes, we chose to associate events coming Wiimote2.Dot1y are the coordinates of the second one. from the Wiimote via Bluetooth, to the mouse movements Any change of the relative distance between the two and to some couples of keys which are typically used on external lights groups can be associated to the shortcuts for graphics application to change the size of brushes (and changing the brushes size. In particular, it has been tested aerographs). with the couple “alt and +/-” as in TheGimp, and the couple By exploiting the features of GlovePie, we firstly defined “shift and ]/[ ” as in Adobe Photoshop. some global variables to detect the screen resolution and the actual state of the detected light sources. Such an operation is necessary to compute the position of the virtual aerosol F. System usage and limitations bomb on the screen and to iteratively control the lights. The Once the switch on its bottom has been turned on and midpoint position and relative distance of the two external GlovePie has been launched with our script, the virtual groups of leds are used to set the actual position of the spray aerosol bomb can be used to move the cursor (sprinkling) on can and the distance (i.e. the size) of it from the screen any image canvas inside a graphics application. By keeping whenever the sprinkling is done. The central group of lights pressed the actuator, users can trace their spray paint on the is used to simulate the sprinkling, once they are switched on screen. by pressing the actuator. The virtual tool has been tested with some widespread graphics applications: first of all with MsPaint, which is released together with Microsoft Windows. Due to the lack var.ScreenW = Screen.DesktopWidth; of shortcuts or other direct ways to change the sprinkling var.ScreenH = Screen.DesktopHeigth; … size, simulation of drawing is severely limited. By utilizing if wiimote.dot1vis then Adobe Photoshop, after setting up the associated shortcuts var.detect=1 (which are “[“ and “]”) to the relative distance between end if external lights, the system is effectively usable. Also with if wiimote.dot2vis then var.detect=2 TheGimp, where once we have set up the keys for simulate end if closeness and remoteness, actions appear to be likely to if wiimote.dot3vis then reality. Despite our efforts to empower the light emission and var.detect=3 adjust the Wiimote sensibility (lowering the threshold of end if detection for each single point), wide angle variations as Code 1. Resolution parameters setting and lights switch control code regards to the Wiimote, or interferences coming from fragment. external sources of light, condition the system. The similarity

of our object to the real one has involved users much more [5] Saffer D., Designing Gestural Interfaces Touchscreens and Interactive than in using mice or graphics tables to spray draw, even if Devices (2008) O’reilly ISBN 10:0-596-51839-0. users are forced to keep the can to the Wiimote and this [6] Diana C., “A Kiss Is Just a Kiss; A Sigh Is Just a Deselection: A Review of Designing Gestural Interfaces”. Interactions Volume hampers the natural movement. This constrain is due to the 16, Issue 1 pp. 45-47 January-February 2009. change of distance between external light, which is used to [7] Sutherland, I. E. “Sketchpad: A man-machine graphical detect shortcuts in order to adapt the brushes size. communication system” in Proceedings of the AFIPS Spring Joint Computer Conference pp. 329–346. 1963.

[8] Buxton, W., Fiume, E., Hill, R., Lee, A. & Woo, C. “Continuous IV. CONCLUSIONS AND FUTURE WORK hand-gesture driven input” in Proceedings of Graphics Interface ’83, Our system has been designed and built to provide users 9th Conference of the Canadian Man-Computer Communications Society. pp. 191–195. 1983. with a tangible object, hand-in-glove as regards to the real [9] Cohen, P. R., et al. “Quickset: multimodal interaction for distributed one, to simulate a common activity like graffiti drawing. applications” in Proceedings of the fifth ACM international Low cost technologies coming from consumer electronics for conference on Multimedia. ACM Press, pp. 31–40. 1997. entertainment made such a work possible, together with the [10] Sturman, D. J., Zeltzer, D., Pieper, S. “Hands-on interaction with availability of software tools and documentation by a wide virtual environments” in Proceedings of the 2nd annual ACM community. Despite its simplicity the Nintendo console SIGGRAPH symposium on User interface software and technology. remote controller has shown to be an appreciable tool for ACM Press pp. 19–24. 1989. effective assessments. Users which have tested the prototype [11] Wexelblat, A. “An approach to natural gesture in virtual environments” ACM Trans. Comput.-Hum. Interact. 2(3) pp. 179– system have confirmed the importance of likelihood of 200. 1995. virtual tools to reality for a much more involving experience. [12] Quek, F. K. H. “Toward a vision-based hand gesture interface” in At the time we are writing we are conducting usability tests Proceedings of the conference on Virtual reality software and with users (through cognitive walkthrough and technology. World Scientific Publishing Co., Inc., pp. 17–31. 1994. questionnaires). They will provide data about better [13] Freeman, W., Weissman, C. D., “Television control by hand involvement with realistic tools, performance, accuracy and gestures” Tech. rep., IEEE Intl. Wkshp. on Automatic Face and precision. Gesture Recognition. 1995. Other experiments have been made and are in progress: [14] Karam, M. PhD Thesis: “A framework for research and design of from a virtual xylophone, to a glove for haptic interfacing. gesture-based human-computer interactions. PhD thesis, University of Southampton 2006.

[15] Moyle, M. & Cockburn, A. “Gesture navigation: an alternative ’back’ for the future” in Proceedings of CHI ’02: CHI ’02 extended abstracts REFERENCES on Human factors in computing systems. ACM Press, New York, [1] Voth, D. “Evolution in gaming” Pervasive Computing, IEEE NY, USA, pp. 822–823. 2002. Volume 6, Issue 2 pp. 7-10. April-June 2007. [16] Ou, J., Fussell, S. R., Chen, X., Setlock, L. D., Yang, J. “Gestural [2] Leder, R.S. et al. “Nintendo Wii remote for computer simulated arm communication over video stream: supporting multimodal interaction and wrist therapy in stroke survivors with upper extremity for remote collaborative physical tasks” in Proceedings of the 5th hemipariesis”, in Proceedings of Virtual Rehabilitation, 2008 pp. 74- international conference on Multimodal interfaces. ACM Press, pp. 74, 25-27 August 2008. 242–249. 2003. [3] Spencer, S. J., et al “A Low Cost Parallel Robot and Trajectory [17] Goza, S. M., Ambrose, R. O., Diftler, M. A., Spain, I. M. Optimization Method for Wrist and Forearm Rehabilitation using the “Telepresence control of the nasa/darpa robonaut on a mobility Wii”, in Proceedings of Biomedical Robotics and Biomechatronics, platform” in Proceedings of the 2004 conference on Human factors in 2008. BioRob 2008. 2nd IEEE RAS & EMBS International computing systems. ACM Press, pp. 623–629. 2004. Conference pp. 869-874. 19-22 Oct. 2008. [18] Fuentes, O. Nelson, R.C. “The virtual tool approach to dextrous [4] Lee, J.C. “Hacking the Nintendo Wii Remote” Pervasive Computing, telemanipulation”, Robotics and Automation, 1996. Proceedings., IEEE Volume 7, Issue 3, pp. 39 – 45 July-Sept. 2008. 1996 IEEE International Conference, Publication Date: 22-28 April 1996, Volume: 2 On page(s): 1700 - 1705 vol.2.