VRMController: An for navigation activities in environments

Hai-Ning Liang*, Yuwei Shi, Feiyu Lu, Jizhou Yang, Konstantinos Papangelis Xi’an Jiaotong-Liverpool University, Suzhou, China

(a) (b) (C) (d) (e) Figure 1. (a-d) Screenshots of the VRMController: (a) the main interface implemented on a mobile device; (b) moving the red dot allows going forward, backward, left and right; (c-d) turning the phone right/left to turn the viewport right/left as if the body of the person is turning; (e) the experimental virtual reality environment simulating a 3-floor shopping mall.

Abstract

Despite the rapid advancement of display capabilities of VR, in the form of wearable goggles like the Oculus for example, there CCS Concept: • Human-centered computing~Touch has been relatively limited progress in the development of input screens • Human-centered computing~Gestural input devices for this technology. In this paper, we describe an input controller that is aimed at supporting users’ navigation activities 1 Introduction in virtual reality environments. Navigation is common in VR environments. The traditional controller for consoles is still Virtual reality (VR) is an emergent technology. Its visual quality a common choice but only now companies are just beginning to has advanced rapidly in recent years. There are now many types introduce new concepts (for example the HTC Vive Controller). of headsets with high resolution displays on the market. In In this research we explore the development of an alternative contrast, development of input devices to allow users to input controller to support users’ navigation activities. This manipulate VR content still lags behind. Navigation or moving process has led to the design and creation of VRMController, an around within a VR environment is one of most common input device based on a designed specifically to activities users do. This can be typically seen in that support single-hand interaction within VR environments. The simulate 3D environments. The default input device for VR design VRMController is based on the results of an initial study headsets is still based on controllers for game consoles. Figure 2 comparing three input devices: an game controller, the HTC (Bottom, Left) is an example of a commercial headset with a Vive controller, and a tablet device. Based on feedback from controller that is based on a game controller. Other commercial participants about the useful features of the three types of devices, products, such as the HTC Vive (Figure 2, Right) and Oculus we have distilled five design guidelines and used them to inform RIFT, have their own concept input device designed from ground- the development of VRMController. The results of a second study up aimed specifically at VR technologies. comparing our controller with an shows that with our controller participants are able to achieve better performance and find that it is easier to use and more usable.

Keywords: input devices, virtual environment, navigation interaction, interface design, mobile device, usability study

Figure 2. (LEFT) The PICO commercial VR headset with its *e-mail: [email protected] Permission to make digital or hard copies of all or part of this work for dedicated controller based on game consoles (see also personal or classroom use is granted without fee provided that copies are http://www.picovr.com/neo.html). (RIGHT) The HTC Vive with its not made or distributed for profit or commercial advantage and that copies specialized two handed controller and motion tracking sensors bear this notice and the full citation on the first page. Copyrights for (see also https://www.vive.com/us/product/). components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to The game controller has been adopted because there are not many post on servers or to redistribute to lists, requires prior specific permission alternatives. Also there is an assumption that interaction with and/or a fee. Request permissions from [email protected]. content in VR headsets is similar to interacting with content VRCAI '16, December 03-04, 2016, Zhuhai, China shown in conventional displays. The main aim of this research is © 2016 ACM. ISBN 978-1-4503-4692-4/16/12…$15.00 DOI: http://dx.doi.org/10.1145/3013971.3014005 to develop an input device to support users’ navigation activities within VR environments. The outcome of the process is VRMController (short for Virtual Reality Mobile Controller), a

455 mobile phone-based device that allows ambidextrous single hand al., 2006; Tan et al., 2006]. Besides these devices, mobile devices interaction. Figures 1a-d show the main interface of the controller have been touted as possible input controller for a variety of and how it can be used with one hand. We tested the controller interactive systems [Pering et al., 2005; Ballagas et al., 2006]. within a VR environment that simulates a shopping mall with Researchers have introduced a variety of implementations based three floors. The design of VRMController is based on a first on using touch-enable mobile devices as controllers (e.g., see study comparing a game controller, the Vive controller, and a [Boring et al., 2007; Dearman et al., 2009; Katzakis and Hori, tablet mobile device. Based on performance results and feedback 2009, 2010; Jeon et al., 2010; McCallum and Irani, 2009; Liang et from participants, we took the features of the three to design al., 2013]). These however are not conducted in the context of VRMController. We then did a second study comparing it with navigation activities but primarily as input devices for broad the traditional game controller. Results show that it has better time tasks, such as manipulating 3D objects (e.g., [Katzakis and Hori, performance than the game controller and that participants find it 2009, 2010]). often more usable. Du et al. [2011] have proposed the use of the tilt and touch The contributions of this paper include: (1) an examination of capabilities of a mobile to enable 3D interaction. Their research three types of input devices for navigation activities in VR indicates that by leveraging the capabilities of mobile devices, a environments; (2) a set of design criteria to aid designers; and (3) suitable controller to support navigation activities can be the introduction of an alternative input device to support these developed. They provide a scenario of using interface on a mobile activities. as a controller for flying around a 3D world. Their research, however, do not provide any usability analysis and comparative In the following sections, we described work related to this evaluation of their interface. Furthermore, their work has not been research. The next two sections will present the two experiments. based on VR systems. Finally, we present the conclusions at the end. Another advantage of an interface based on mobile devices is that 2 Related work it can easily be modified and adapted to meet the specific needs of users. For example, buttons can be positioned and their size In this section we will discuss two aspects related to this research. adjusted based on the size of users’ hands. The adaptability is not The first aspect is about prior research on navigation within VR possible in other physical devices such as game controllers or the environments. This is followed by a discussion about input Oculus Touch. devices for interacting with VR content. Finally, the research goal for this project is described in more detail. 2.3 Research goal

Given the lack of research on the suitability of input devices for 2.1 Navigation in VR supporting navigation activities using VR headsets, this research is an attempt to fill this gap. To achieve this, we run an initial 3D Navigation has been studied in traditional TV/ exploratory study of 3 types of input devices. The first type is displays and CAVE-type of 3D environments [Chen, 2004; Tan et what is normally chosen to use: A controller for game consoles al., 2006; Cliburn et al., 2007]. Research suggests that immersion such as the Xbox and PlayStation. The second type represents a is more conducive to enhanced experiences navigating 3D spaces new class of controllers such as the HTC Vive and Oculus Touch in 2D screens. For example, Tan et al. [15], from a series of (see Figure 2 above). These two devices rely on both hands studies involving users in navigating 3D spaces, reported several holding a device that is tracked by external sensors. At the time of advantages of using large displays as compared to smaller, this research, only the HTC Vive was available in the market so desktop size screens. Due to the bigger size of the displays, users we chose it to explore its suitability for navigation activities. The had a greater sense of immersion and that helped them develop third type is a virtual controller that is implemented on a mobile better cognitive maps of the virtual worlds, which subsequently device (in the first experiment a tablet) because we wanted to led users to perform better in their navigation tasks in 3D (also simulate the functions of the game controller to assess if people supported by [Bakdash et al., 2006]). VR headsets are intended to will find it possible to use. Although we had hypothesized that the provide an even higher level of immersion and as such are thought Vive and tablet will underperform the game controller, we wanted to be suitable means to support navigating 3D virtual spaces. to conduct the experiment in an exploratory manner in order to distill useful features of these three types of devices. This process 2.2 Input devices for VR interaction led to development of five design guidelines which were then used to guide the development of VRMController. In our second Navigation in VR is concerned with the movement (or change in study we compared it with the game controller. the viewpoints) that occurs within a simulation of a 3D physical environment (e.g., museum or library) [Liang and Sedig, 2009]. When examined more closely, this movement can be divided into 3 Experiment 1: Examination of 3 types of three related activities. The first two are about changing viewing input devices perspectives—this is the equivalent of two actions: (1) changing the head and (2) turning the body in different directions. The third This first study investigated the performance and usability of 3 activity is about changing locations within the environment—this types of input devices to support navigation activities within VR is what happens when people are walking around. environments. The three devices that were chosen represent three generations of such devices: the Xbox game controller, the HTC A suitable input device should allow users to have control of how Vive, and a tablet device. We focused on comparing their these three activities can be performed. In their experiments of 3D performance and subjective view of these devices. navigation tasks, researchers have often relied on the use of devices such a mouse and keyboard; a gaming or controller; or motion-tracked wands [Bakdash et al., 2006; Ni et

456 3.1 Apparatus and tasks The main task of this experiment is to walk around a 3-floor shopping mall simulated as a 3D environment and collect 7 items Figure 3 shows the three input devices. The Xbox Controller placed in different locations. Figure 1e above and Figure 5 below (Figure 3a) has two sticks, one for each thumb, a directional pad show two screenshots of the mall. The environment was set to be for the left thumb and a number of buttons. We mainly used the dark, but still visible. The idea was for participants to be able to sticks and the directional pad to allow moving in the VR locate the items with more ease as they are highlighted. We environment. Users can use the left stick or pad for walking and allowed the option of turning on/off a spotlight or flashlight the right stick to turn their body left or right. (represented as a brighter area in the middle). Moving their heads with turn the spotlight in the corresponding direction.

(a) (b) (c) Figure 3.The three types of controllers used in the first experiment: (a) the Xbox Controller; (b) the HTC Vive Dual- Handle Controller; (c) our in-house built tablet interface.

The HTC Vive (Figure 3b) is one of most advanced controllers for VR headsets. The controller consists of two handles, one for each hand. Each handle has a round trackpad, a dual-stage trigger with haptic feedback, and three buttons (see Figure 4a). The handles Figure 5. A view of the shopping mall. The number (in this case and the headset are tracked in real time with two sensing stations 64) represents how much time participants still has to find all the located around the user (see Figure 4b). Similar to the Xbox remaining targets. The bright middle circle is the spotlight or Controller walking movements is controlled with the left flashlight that can be turned on/off and on based on users’ preferences. Turning their head will turn the spotlight. The small trackpad, while turning left/right with the right trackpad. squares under the time indicator are the items already found and

to be located still.

3.2 Participants and procedure

Twelve participants (9 males, 3 females) in total participated in the experiment. They were all students at a local university with an age range of 21-28. Their participation was voluntary.

At the beginning of the experiments, participants were asked to complete a questionnaire to collect demographic data and their past experience with VR systems. Then, they were given time to Figure 4. The set up for the HTC Vive environment. (LEFT) A familiarize themselves with the three input devices. After, they participant interacting with the system by holding the two handles. (RIGHT) The button to switch on/off the flashlight. were asked to complete the experiment using one device at the time. For each device, all 7 targets were placed in the same locations—but their locations differed from device to device to The Tablet Controller (Figure 3c) is a Shield gaming avoid participants being able to remember their locations. The tablet with an 8 inch 1920*1200 resolution IPS touch screen order of devices were decided using a Latin Square design to running on Android. The device is connected to a avoid carry over effects. After they completed the experiment, receiver attached to an Arduino board which is then attached to a participants were asked to complete a short questionnaire to desktop computer. The interface consists of two circlers. The left collect subjective feedback on the devices. In addition, we circle is from controlling walking movements while the right interviewed them to elicit features that they would like to have in circle is for turning around. This design closely resembles what a controller for navigation activities. two sticks on the Xbox controller looks like. To give some tactile feedback, we placed a piece of plastic screen cover on top of each 3.4 Results and discussion circle. With the Game Controller, participants took an average of 36.5 For the Xbox Controller and Tablet Controller, we used the seconds to find all 7 targets. This was followed by Tablet CV1 headset and used the HTC headset with the Controller with 42.50 seconds. Finally, with the Vive Controller, Vive Controller. Both the Oculus RIFT and Vive have a sensor it took participants the longest with an average of 63.95 seconds. that tracks the head movements to allow changing the viewing It was not surprise that the Game Controller had the best perspective based on head movements. The experiment was run performance due to users’ familiarity with it. However, it was on computer with i7 with 4 GHz, dedicated GTX 1070 graphics surprising to see that the Vive Controller led to the worst time of card, and 32 GB memory. the three. This controller was basically the same as the Game Controller with almost the same amount of tactile feedback. On the other hand, we were taken aback by the performance of the

457 Tablet Controller, especially it being so close to the Game of VRMController. Controller. 4.1 Apparatus and tasks Data collected in the post-experiment interviews indicated that the Vive Controller was the most interesting but not easy to use due We used the same Xbox controller in the previous experiment. to the need to use both hands. Participants commented that they The VRMController was implemented on a Google Nexus 5x had difficulty manipulating “two controllers” at the same time. phone with a Qualcomm Snapdragon 8081.8 GHz , 5.2 That is, participants thought they were two, rather than one, single inches (1920×1080) LCD touch screen, and a size of input device. They indicated that they prefer to have one single 147.0×72.6×7.9 mm. Extra layers of screen protector were placed device small enough to fit into their hands. In addition, they felt on the center of the screen to indicate the boundary of the middle that they would need a lot of training to feel that they can use the circle and to provide tactile feedback to users. The circle was Vive Controller in an effective manner. They said that long meant for users to walk around the environment. To turn left or training time would be a discouraging factor to choose the device right, users needed to only flip the device. Figure 1 (a-d) shows in the future. the final design of VRMController and how it can be used.

Although the Game Controller was their first choice in the future VRMController meets the 5 design criteria. It uses a very light to do navigation activities, participants felt that it was not very mobile phone to support one-handed interaction, thereby “engaging” to use it for simple navigations activities. Also they minimizing device and physical constraints (DC1+DC2). Second, mentioned that for this kind of activities a one-hand device would users can use it with their left or right hand. In addition, they can be sufficient because holding the device with two hands was tiring make the circle bigger/smaller and/or closer to the top/bottom or and not very comfortable, especially given that their normal way left/right. These two design aspects are inspired by the to interact with the VR system was standing. LenseMouse [Yang et al., 2010] and support user adaptation of the controller (DC3). Third, the device is meant to be easy and Participants were somewhat split about the Tablet Controller. intuitive to use (DC4). We chose a mobile device because users Some participants said that it was a good device to use; while would be familiar with how to use its touch screen and tilt others said that it was not easy to use. Similar to the Game functions. In addition, the device is meant to be comfortable to Controller, this device was uncomfortable to hold for a long time use (DC5). To achieve this, we got inspiration from design while standing. Some participants gave positive feedback about guidelines for tilt interaction with mobile phones [Rahman et al., the ability to change the size of the circle and move it around to 2009] such that the angles are easy to perform. Finally, its design suit their needs. is aimed to maximize performance (DC5).

Based on the results of the first experiment and users’ feedback, As distilled from the results of experiment one, we observed that an ideal input device to support navigation activities in VR it is cognitive easier for users to simply use one hand to carry environments should have the following five design criteria. actions—rather than two. In addition, we purposely included two distinct types of motions that were based on two parts of the arm:  DC1: Minimal device constraints. The device should not be (1) thumb movements to allow walking around the environment; too cumbersome but needs to be simple, light, and easy to (2) wrist tilt motions to let users turn right/left. One reason why hold in users’ hands. we chose a small form factor mobile phone as the platform for the  DC2: Minimal physical constraints. The device should not controller was because people often would interact with it require users to exert much physical effort. In addition, it standing or sometimes even walking. Given that when wearing should minimize the need for dual-hand if single handed VR headsets people would also be standing, the phone could interaction is sufficient. transition positively into a controller.  DC3: Allow user adaptation. Input devices are often personal items which users tend to customize to make them more To assess whether VRMController would meet these 5 design fitting to their physical needs and personal preferences. Any criteria, we conducted a second experiment in which we new devices should allow users to change certain parameters. compared it against the baseline Game Controller. The apparatus  DC4: Minimize the need for extended training periods. The used is the same as in the first experiments. The Oculus RIFT device should be usable and easy to use, without the need for CV1 headset was used in conjunction with either VRMController prolonged periods of training. or an Xbox Game Controller. The same desktop computer was  DC5: Maximize performance. The device should have better also used. or at least comparable performance with existing devices. Participants were requested to do the same task as in the first Based on the lessons of the first experiment, we went on to design experiment but instead of locating 7 items they had to locate 10 and develop VRMController following the above design criteria items with each device by navigating in the same VR environment and conducted a second experiment. of the 3-floor shopping mall. The items were placed in the same locations for each device. The order of the devices were alternated 4 Experiment 2: VRMController vs game using a Latin Square design. controller 4.2 Participants and procedure In this experiment, we compared VRMController with the Xbox game controller because it was the best performing device from 16 participants (5 males, 11 females), aged 17 to 22 years old, the first experiment. The main aim is to see if the two have were involved in this experiment. They were all undergraduate comparable performance and also to investigate the usability level students at a local university. None of them participated in the first experiment.

458 Acknowledgements As with experiment one, participants completed a short questionnaire to collect information about demographics and This project is partially supported by the XJTLU Summer previous experiences with VR. Then they were given time to get Undergraduate Research Fellowship and Crenovator Lab familiar with the two devices, the VR headset and environment. (CrenoLab) Suzhou. The authors would also like to thank Simon After this, they would proceed to do the experiment. At the end, Chang and Peter (Shun) Yao from CrenoLab for their input. they completed a short questionnaire to collect their subjective preferences of the two devices.

4.3 Results and discussion References

In terms of time performance, participants did better using the BAKDASH, J.Z., AUGUSTYN, J.S., AND PROFFITT, D.R. 2006. Large displays enhance spatial knowledge of a virtual environment. IN VRMController than Game Controller, averaging 16.26 seconds Proceedings of APGV’06, 56-173. compared to 22.05 seconds—a significant difference between the BALLAGAS, R., ROHS, M., SHERIDAN, J. AND BORCHERS, J. 2006. two. Data from the post experiment questionnaire indicated that it The Smart Phone: A Ubiquitous Input Device. IEEE Pervasive was their preferred device. Of the 16 participants, 9 thought that Computing, vol.5, no.1, 70-77. VRMController is more effective. 12 of them said that it was BORING, S., ALTENDORFER, M., BROLL, G., HILLIGES, O., AND more usable than the Game Controller and liked the fact that it BUTZ, A. 2007. Shoot & Copy: Phonecam-based information was based on a familiar device. Interestingly, 8 participants said transfer from public displays onto mobile phones. In that the Game Controller gave them more control but most (10) Proceedings of Mobility '07, 24-31. said the VRMController was more comfortable to use, while 12 CHEN, C. 2004. Information visualization: Beyond the horizon. told us that it was intuitive to use and they could use it with very SPRINGER. little prior experience and knowledge. On the other hand, some CLIBURN, D., WINLOCK, T., RILEA, S., AND VAN DONSEL, M. 2007. participants commented that the Game Controller gave them a Dynamic landmark placement as a navigation aid in virtual sense of “gaming”. Most participants (12) agreed that the worlds. In Proceedings of the 2007 ACM symposium on Virtual VRMController was a better device overall if they had to stand reality software and technology (VRST '07), 211-214. while interacting with the content from the VR headset. One DOI=http://dx.doi.org.liverpool.idm.oclc.org/10.1145/1315184. element that stood out for most participants was the fact that it 1315225. was one-handed and that they could switch hands without the DEARMAN, D. AND TRUONG, K.N. 2009. BlueTone: A framework need to stop their current activity and do some kind of set up. This for interacting with public displays using dual-tone multi- would not be possible with the Game Controller, which after some frequency through Bluetooth. In Proceedings of time interacting with it, caused some discomfort (6 females UBUCOMP'09, 97-100. indicated they got somewhat tired near the end of the study with DU, Y., REN, H., PAN, G. AND LI, S. 2011. Tilt & touch: mobile the Game Controller). phone for 3D interaction. In Proceedings of the 13th ACM International Conference on Ubiquitous Computing (UbiComp '11). ACM, New York, NY, USA, pp. 485-486. 5 Conclusions DOI=http://dx.doi.org.liverpool.idm.oclc.org/10.1145/2030112. 2030183 JEON, S., HWANG, J., KIM, G.J., AND BILLINGHURST, M. 2010. In this paper we reported our explorations in the development of Interaction with large ubiquitous displays using camera- an input device that can support users’ navigation activities in VR equipped mobile phones. Pervasive Ubiquitous Computing, 14, environments such as the Oculus RIFT or HTC Vive. To this end, 83-94. we have conducted an exploratory study with three generation of KATZAKIS, N. AND HORI, M. 2009. Mobile Phones as 3-DOF input devices: a game controller, the HTC Vive, and a touch- Controllers: A Comparative Study. In Proceedings of IEEE enabled tablet device. From this study five design guidelines are International Symposium on Dependable, Autonomic and extrapolated: Minimal device constraints; minimal physical Secure Computing, Chengdu, China, Dec. 12-14, pp. 345-349. constraints; user adaption; minimal training needed; and KATZAKIS, N. AND HORI, M. 2010. Mobile devices as multi-DOF maximum performance. We have used these guidelines to inform controllers. In Proceedings of IEEE Symposium on 3D User the design of VRMController, which is a mobile phone based Interfaces (3DUI), pp. 139-140. device that support ambidextrous, single hand interaction. In LEE, J., KIM, B., SUH, B., AND KOH, E. 2016. Exploring the Front comparison with devices that require two hands, VRMController Touch Interface for Virtual Reality Headsets. In Proceedings of is meant to be a versatile, simple to use, and light. The results of a the 2016 CHI Conference Extended Abstracts on Human second study comparing the VRMController with a game Factors in Computing Systems (CHI EA '16), 2585-2591. DOI: controller show that participants are able to perform faster in http://dx.doi.org.liverpool.idm.oclc.org/10.1145/2851581.2892 locating objects in the 3D environment with our device. In 344. addition, their subjective feedback strongly indicate that they have LIANG, H.-N., WILLIAMS, C., STUERZLINGER, W., SEMEGEN, M., found it comfortable, easy, and pleasant to use. All in all, the AND IRANI, P. 2013. An investigation of suitable interactions for VRMController is an effective and usable device to use when 3D manipulation of distant objects through a mobile device. dealing with navigation activities in VR environments. International Journal of Innovative Computing, Information and Control, 9(12, December 2013), pp.4737-4752. LIANG, H-N AND SEDIG, K. 2009. Characterizing navigation in interactive learning environments. Interactive Learning Environments, 17(1), PP. 53-75. MCCALLUM, D.C. AND IRANI, P. 2009. ARC-Pad: absolute+relative cursor positioning for large displays with a

459 mobile . In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (UIST '09), 153-156. NI, T., BOWMAN, D.A., AND CHEN, J. 2006. Increased Display Size and Resolution Improve Task Performance in Information-Rich Virtual Environments. In Proceedings of GI’06, 139-146. PERING, T., BALLAGAS, R. AND WANT, R. 2005. Spontaneous marriages of mobile devices and interactive spaces. Communications of the ACM, vol.48, no.9, 53-59. RAHMAN, M., GUSTAFSON, S., IRANI, P. AND SUBRAMANIAN, S. 2009. Tilt Techniques: Investigating the Dexterity of Wrist- based Input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '09), 1943-1952. TAN, D.S., GERGLE, D., SCUPELLI, P., AND PAUSCH, R. 2006. Physically Large Displays Improve Performance on Spatial Tasks. ACM Trans. HCI, 13(1), pp.71-99. YANG, X.-D., MAK, E., MCCALLUM, D.C., IRANI, P., CAO, X. AND IZADI, S. 2010. LensMouse: Augmenting the Mouse with an Interactive Touch Display. In Proceedings of the 28th international conference on Human factors in computing systems (CHI '10), Atlanta, Georgia, USA. ACM, 2431-2440.

460