360 VR Based Robot Teleoperation Interface for Virtual Tour
Total Page:16
File Type:pdf, Size:1020Kb
360 VR Based Robot Teleoperation Interface for Virtual Tour Yeonju Oh Ramviyas Parasuraman Department of Computer and Information Technology Department of Computer and Information Technology Purdue University Purdue University West Lafayette, IN 47907, USA West Lafayette, IN 47907, USA [email protected] [email protected] Tim McGraw Byung-Cheol Min Department of Computer and Graphics Technology Department of Computer and Information Technology Purdue University Purdue University West Lafayette, IN 47907, USA West Lafayette, IN 47907, USA [email protected] [email protected] ABSTRACT VR Display We propose a novel mobile robot teleoperation interface that demon- Remote Environment strates the applicability of robot-aided remote telepresence system 360⁰ Camera with a virtual reality (VR) device in a virtual tour scenario. To im- prove the realism and provide an intuitive replica of the remote Live Streaming Directional environment at the user interface, we implemented a system that Antenna automatically moves a mobile robot (viewpoint) while displaying a 360-degree live video streamed from the robot on a virtual reality User Input Mobile Robot gear (Oculus Rift). Once the user chooses a destination location User among a given set of options, the robot generates a route to the des- tination based on a shortest path graph, and travels along the route Figure 1: Illustrative example of the proposed VR interface. using a wireless signal tracking method that depends on measuring the Direction of Arrival (DOA) of radio signal. In this paper, we Therefore, to mitigate such limitations and to realize a true im- present an overview of the system and the architecture, highlight mersive remote tour experience, this study proposes a system ex- the current progress in the system development, and discuss the ploiting a VR display that produces a (near real-time) stream of implementation aspects of the above system. the live 360-degree camera from a moving agent, specifically a mo- bile robot. Figure1 illustrates an example scenario of the proposed 1 INTRODUCTION system. The VR display shows the general interface so that the In addition to the entertainment industry, Virtual Reality (VR) has user who is equipped with a VR device can visualize the robot’s a myriad of potential application areas such as tourism [22], indus- 3D view and also control the robot to navigate through the remote trial design [6], architecture, real estate, medical care [7, 8], and environment (and change the viewpoint). education [20]. Among these applications, we are interested in the Recent years have witnessed active research in robot control use of VR in a virtual tour scenario, where users can be provided using VR [4]. Particularly, immersive teleoperation systems us- with vivid touring experiences of real-world environments. For ing Head Mounted Displays (HMD) have been developed [10, 12]. instance, in the 2017 Google I/O conference, Google introduced the Telepresence via the VR interface has shown to be easier and more motif of educational VR content using smartphones, with the main intuitive than interfaces using liquid crystal displays and joysticks feature being that students can watch 360-degree field trip videos [3, 11]. For instance, in [1], a panoramic image is generated by join- to realistically learn what they see in a book. Through the pilot ing the images taken by the omnidirectional camera on a Remotely studies, students could engage in more immersive, interactive, and Operated underwater Vehicle (ROV), and the VR display renders social activities to obtain enhanced learning experiences. the combined image to improve the spatial perception in the under- Currently, such virtual tour applications have limitations mainly water environment. In another study [14], a VR-based teleoperation in terms of displaying a prerecorded media (video or panorama interface is developed to allow users to perform specific tasks (as- images) [9, 13, 21]. In [9, 21], the studies constructed a 3D map sembling work) with a robot manipulator by mimicking the user’s for a virtual tour of world cultural heritage, but they do not re- movements operated with the VR controller, which is more accurate flect real-time situation. In [13], the authors proposed a web-based than the conventional auto assembly used in automated processes. telepresence system with 360-degree panorama image. This mobile These previous studies show that the VR interface can be more robot platform adopts a landmark-based path planning entered by effective than the general display (such as a screen and ajoystick the users. The monitoring window provides an omnidirectional controller) and can be used in real life through an integration of image but does not present it using a VR device. In addition, existing various systems. Taking inspirations from these works, in this paper, research and applications can only adjust the viewpoint only at a we propose a VR interface that incorporates a 360-degree camera fixed position or move using manual control. from a mobile robot to better utilize the immersive experience of User Server 1 Robot combining two fisheye lenses to create a total of 360-degree im- age. Thus, by stitching the image frames like orange guidelines in VR Deliver robot’s info RSSI Directional ROS Bridge Figure 3(a), a live video stream is rendered on a spherical object Interface Antenna User’s destination shown in Figure 3(b). Each fisheye image is presented on each hemi- Oculus Rift • HMD Deliver sensor info Wheels sphere to produce a panoramic image in one perfect sphere. The ROS Nodes • Controller Sensors 3D display the user can see is shown in Figure 3(c) (but stretched Commands for • Path planning navigation to 2D in the picture). Since each lens takes more than 180 degrees • RSSI tracking 360 of photographs, when cutting and joining each sphere, a part of Camera Server 2 the original photograph is cut and merged. At this time, a slight boundary is created in the panoramic photograph. To improve the Video stream : Network related component quality of streaming in terms of latency, we have to change the HTTP server : Navigation related component streaming settings such as reducing the bitrate of the video frame • 360‐degree video feed : Independent component or streaming buffer size, because the camera needs to send two photos together, which is a feature of dual fisheye lenses. Due to Figure 2: An overview of the proposed 360 VR based virtual this limitations, the changes of settings may degrade the quality (remote) tour system. of video. Therefore, there is a trade-off every time we do this post processing. the VR display. Additionally, we design an autonomous navigation 2.2 Telepresence Robot Server system for the robot through a wireless signal tracking method Once the users select their destinations using the UI, the robot [15, 16]. This way, the users are expected to feel less fatigue (and starts to plan the path based on the registered Wi-Fi Access Points not worry about fine tuned movement control) during the virtual (APs) and tracks the generated path using the directional antenna tour experience through the VR display and have a more relaxed based wireless signal tracking method proposed in [15]. Wi-Fi sig- experience by minimizing the mental workload of navigating the nal tracking can take several advantages in terms of cost effective environment. and supporting a consistent live streaming and path planning. By Moreover, we expect that this platform can be generalized to using indoor APs, the server can consistently keep streaming the realize an intuitive telepresence robot with a 360-degree camera live video in a farther space. Also, even though the selected des- video streaming interface in VR environments. Therefore, the us- tination is not in the line of sight, the path can be generated by ability of the VR based robot interface can be improved not only for passing through the waypoint AP where the robot is accessible at the case of virtual tour but also broadly for the general telepresence the current location. case. 2.2.1 Hardware. Multiple APs are pre-installed throughout the 2 METHODOLOGY AND SYSTEM DESIGN environment, and they can serve as a way point or the destination This section provides the overall architecture and discusses the user for the robot’s tracking as illustrated in Figure4. This approach is interface (UI) components for building the entire system. cost-effective, compared to other infrastructures using beacons or RFID. It is also very effective as typical Wi-Fi APs can be utilized. 2.1 System Architecture A 360-degree camera and a rotating Wi-Fi directional antenna are connected to the onboard computer on a mobile robot that To fully support the concepts of the envisioned virtual tour, we de- provides ROS-compatible driver. The directional antenna measures termine the following essential components: a VR video streaming the Wi-Fi Received Signal Strength Indicator (RSSI) within a specific and rendering, robot’s signal tracking for the indoor autonomous angular range rotated by a servo pan system. This angular scan of navigation, and integration with VR interface. Figure2 shows an the RSSI is then used to calculate the Direction of Arrival (DOA) of overview of the overall system architecture. This categorizes the the Wi-Fi AP. The built-in ultrasonic sensors on the mobile robot system into three entities: VR user interface at the client (User), a are used to determine the DOA of the obstacles, which is then used mobile robot which is the moving agent (Robot), and path planning to avoid obstacles on the trajectory. and tracking function to generate the robot path (Server). The robot’s path planning features are integrated with Robot 2.2.2 Software. The software module includes a path planning Operating System (ROS) [19], which is a powerful tool to simplify algorithm, which finds the shortest path from a graph consisting of the development processes in robotic programming.