A Low-Cost Camera-Based Tracking Alternative for Virtual Reality
Total Page:16
File Type:pdf, Size:1020Kb
SBC { Proceedings of SBGames 2018 | ISSN: 2179-2259 Computing Track { Full Papers Controlling First-Person Character Movement: A Low-Cost Camera-based Tracking Alternative for Virtual Reality Rodrigo F. Berriel Thiago M. Paixao˜ Ivo Nicchio, Thiago Oliveira-Santos Departamento de Informatica´ Coordenadoria de Informatica´ Departamento de Informatica´ UFES IFES UFES Vitoria,´ Brazil Serra, Brazil Vitoria,´ Brazil Email: [email protected] Email: [email protected] Email: [email protected], [email protected] Abstract—Virtual reality (VR) gaming is a billion-dollar may also be extended when aided by specific sensors and industry that aims at providing a more realistic and excit- controller devices, however these additional equipment make ing immersive experience in computer-simulated environment. the VR setup costly to ordinary games users. For the Besides hardware capabilities and high-quality 3-D graphics, the feeling of immersion is augmented by incorporating real- intended movement control application, the Oculus Rift istic player-game interactions, including the control of game headset can be empowered with a pair Oculus Touch tracked characters movements based on real/natural movements of the controllers, as in the VR game “Bullet Train” [7]. Also, player and objects. To this intent, modern VR gaming usually the game system (or other VR applications) may allow the leverages virtual reality headsets and additional devices/sensors combination at software level of interfaces from different whose costs are normally expensive for ordinary games users. In this paper, we propose a low-cost VR gaming system in manufacturers. This idea can be seen in [8], where the HMD which the smartphone is used as display and processing units, Oculus Rift head tracker is combined with the Leap Motion, and its rear camera is used as the main sensor for controlling which is used to track hand-based gestures for first person the character movements in a first-person shooter game (FPS) movement control. Oculus Rift in [9] is also combined with developed in this work. The VR game system can operate, Microsoft Kinect 2 for walk-through navigation based on on average, at 31 frames per second. Regarding the usability, 19 of 24 interviewed users evaluated the character movement body and legs tracking. For additional reference, the reader precision as good or very good, while 95.8% considered the may refer to [10]–[12]. gun movement as good or very good. Also, all the users would For cheaper VR equipment, the VR industry has been recommend the movement control system as a real-virtual heavily investing in headsets which allow coupling smart- interaction alternative for VR gaming. phones as processing and display units (e.g., Samsung Gear Keywords-Virtual reality gaming; Character movement con- VR and Google Cardboard). Nevertheless, the smartphone troller; Camera-based tracking; First-person shooter game. processing capabilities and sensors are still underused in the context of VR gaming industry. Particularly, we are I. INTRODUCTION interested in providing low-cost movement control using the Virtual reality (VR) technology provides the user an inter- smartphone built-in camera as a capture device following a active experience in a 3-D computer-generated environment similar approach to that in [13], [14]. To this intent, we and has been increasingly explored in different field of hu- developed a first-person shooter (FPS) game [9], [15] as man activities, ranging from purely entertainment purposes scenery for this investigation. to military ones. Particularly, VR is a trend towards a more Two movement control tasks are included in the scope realistic and exciting immersive gaming experience given of this work: the gun movement handling and the first the various possibilities of human-machine interactions: eye person walk-through navigation. In the context of FPS movements/gestures [1], head gestures [2], and skin contact games, a handheld controller device that resembles a real between people [3], indirect physiological inputs [4], [5] gun (e.g., pistol joystick) is important for an immersive (e.g,. heartbeat, breath), to name a few. According to the feeling experience [16] instead of relying on purely in-air openPR’s report in [6], the VR gaming market is estimated gestures. Also, the shooter character navigation in the game to achieve the quantity of US$ 14.6 billion by the end of can be mapped directly on the player movement across the 2023, which represents 30.5% of Compound Annual Growth real space. The player movements are directly translated into Rate (CAGR) for the period of 2017-2023. the same character movements, i.e., they are not required to Modern VR gaming usually leverages virtual reality head- be decoded to trigger an associated action, as in the human- sets (e.g. Oculus Rift and HTC Vive) that basically include flying metaphor [17]. a stereoscopic head-mounted display (HMD) and a stereo The proposed low-cost VR gaming system is basically sound (headphones). Additionally, VR headsets capabilities composed of four components: a VR BOX headset – one XVII SBGames { Foz do Igua¸cu{ PR { Brazil, October 29th { November 1st, 2018 418 SBC { Proceedings of SBGames 2018 | ISSN: 2179-2259 Computing Track { Full Papers Tracking Targets Detection and Tracking of Pose Mapping Targets Bluetooth Controller Custom Weapon’s Target Wall’s Target Weapon Weapon’s Body’s Bluetooth Trigger Movement Movement Custom Weapon Smartphone Processing and Sensors: Camera Accelerometer Gyroscope Headset Head-Mounted Display Head-Mounted Head’s Character’s Display Movement Movement Figure 1. Overview. The proposed system comprises four components: a headset, a smartphone, a Bluetooth controller, and tracking targets. The player can customize its own weapon by combining a Bluetooth controller with a tracking target. The smartphone attached to the headset is used as a head-mounted display. The camera of the smartphone is used to capture image wherein tracking targets are detected and tracked. The poses of the tracking targets are mapped to their virtual counterparts, enabling the control of the movements in the game. of the cheapest headsets commercially available – , a smart- all the users would recommend the movement control system phone, a Bluetooth controller, and tracking targets. All the as a real-virtual interaction alternative for VR gaming. On processing is performed in the smartphone (which is coupled average, the system can operate in 31 frames per second to the VR BOX), including the movement control handling. (using the Moto X 2). Throughout the game execution, images are captured by the In summary, the main contributions of this work are: smartphone rear camera which points forward in relation A low-cost smartphone-based VR gaming system; • to the player face. These images are processed by the A low-processing movement control scheme imple- • smartphone CPU/GPU using computer vision techniques to mented directly into the smartphone and that can be detect and track the targets (i.e., textured pattern markers applied to a variety of hand-made controllers; made of paper) placed onto the objects of interest: the wall, A first-person shooter game created as a testing • a fixed reference used to estimate the relative movement scenery; of the game character (i.e., shooter’s walk-through); and An usability investigation of the proposed implementa- • the shooter’s pistol, so that its position (i.e., location and tion. pose) can be determined for the sake of gun movement The rest of this paper is organized as follow. The pro- control. The Bluetooth controller, whose shape resembles posed gaming system, focusing on the movement control, to a pistol handle, has the button trigger function. Actually, is described in SectionII. The experimental methodology is the marker-based tracking paradigm can be easily applied present in Section III, and SectionIV discusses the obtained to hand-made controllers (e.g., car steering wheel, flight results. Concluding remarks are made in SectionV. joystick), following the Nintendo Labo’s [18] “build your own controller” philosophy. II. PROPOSED SYSTEM To evaluate the proposed movement control system, a The proposed low-cost alternative for controlling move- first-person shooter game was developed. The usability of ments in Virtual Reality (VR) games comprises four com- the system was evaluated by 24 participants using question ponents: a headset, a smartphone, a bluetooth control, and forms, and it was also analyzed the system time performance tracking targets. The headset is responsible for housing for different smartphone models. The obtained results show the smartphone, and together they are used as a head- that 19 of the interviewed users evaluated the character mounted display (HMD). The smartphone, besides being the movement precision as good or very good, while 95.8% display, is also responsible for processing both the movement considered the gun movement as good or very good. Also, control system and the game itself. The bluetooth control XVII SBGames { Foz do Igua¸cu{ PR { Brazil, October 29th { November 1st, 2018 419 SBC { Proceedings of SBGames 2018 | ISSN: 2179-2259 Computing Track { Full Papers is responsible for triggering actions, such as shooting a game. The latter are enabled by the aforementioned process gun. The tracking targets (for simplicity, we will call them of detection and tracking of the targets attached to fixed targets), patterns printed in a piece of paper, are responsible places (e.g., walls). To enable the movement of the body, a for enabling the alternative movement control system. In the pre-defined position in the virtual environment is linked to a proposed system, the sensors of the smartphone are used to given target. Once detected, this target is assumed to be static help controlling the character in the game environment. For and treated as a virtual anchor, i.e., when the player moves instance, with the help of computer vision techniques, the away from the target in one direction, this movements is built-in camera is used to detect and track the targets.