The Development of a Novel Eye Tracking Based Remote Camera Control for Mining Teleoperation
Total Page：16
File Type：pdf, Size：1020Kb
Seventh Large Open Pit Mining Conference 2010 The Development of a Novel Eye Tracking Based Remote Camera Control for Mining Teleoperation D Zhu, T Gedeon and K Taylor Reference Number: 44 Contact Author: (Use “Author Details” style) Full name: Dingyun Zhu Position title: PhD Candidate Organisation Name: CSIRO ICT Centre & School of Computer Science, ANU Address: CS&IT Building, Australian National University, North Road Acton ACT 0200, Canberra Phone: +61 2 6216 7141 Fax: +61 2 6216 7111 Mobile: 0420905362 Email: [email protected] 1 The Development of a Novel Eye Tracking Based Remote Camera Control for Mining Teleoperation D Zhu 1, T Gedeon 2 and K Taylor 3 1. Dingyun Zhu PhD Candidate, CSIRO ICT Centre & School of Computer Science, ANU, CS&IT Building, Australian National University, North Road Acton ACT 0200, Canberra. [email protected] 2. Tom Gedeon Professor, School of Computer Science, ANU, CS&IT Building, Australian National University, North Road Acton ACT 0200, Canberra. [email protected] 3. Ken Taylor Research Scientist, CSIRO ICT Centre, CS&IT Building, Australian National University, North Road Acton ACT 0200, Canberra. [email protected] 2 ABSTRACT The use of teleoperation in mining is increasing due to requirements to improve safety and reduce the number of people required to work in remote and difficult environments. CSIRO’s recent collaborative project with Rio Tinto was to develop a teleoperated control system to the primary rock breaker at the West Angelas iron ore mine in Western Australia, with the objective of demonstrating the feasibility of remote rock breaking over long distances. This required controlling both the rock breaker and an array of pan-tilt-zoom (PTZ) cameras using joysticks. Controlling the cameras distracted operators from the primary task of controlling the rock breaker to position and fire the hammer. This lengthened task times as both the operator’s attention and their hands have to swap between different control interfaces. In this paper, we present a novel design using human eye gaze as an interactive input for the remote camera control. It follows a simple and natural design principle: “Whatever you look at the screen, it goes to the centre. ” A prototype system has been implemented by integrating computer vision based eye tracking technology, with the advantage of being real-time, robust, and marker-less. We conducted a user evaluation, where users undertook a control task while obscuring the direct view of the working area and using a remote camera to transfer the live video stream as the visual feedback. Subjects were required to participate in the experiment by using eye tracking camera control and traditional joystick control respectively. The statistical analysis of objective measures indicated that our novel eye tracking control significantly outperformed the joystick control. The results of the post-experimental subjective measures also revealed much higher user preference for using eye tracking control, which provides clear evidence that this design is a better interface for remote camera control in mining teleoperation settings. Keywords: mining teleoperation, eye tracking, remote camera control, rock breaking, user interface, usability evaluation INTRODUCTION Safety is always the most important issue in any industry and the mining industry is no different. In order to prevent people from working under hazardous conditions or in difficult environments, teleoperation has been increasingly used as an effective solution in mining industries. Compared to the common line-of-sight control on the mine sites, such a solution allows people to remotely control or manipulate complex mining machinery over long distances, with the merits of providing replaceable surrogates, reducing number of staff and being cost-effective. Although there has been a continuous long-term effort spent on researching and developing autonomous or semi-autonomous systems for a variety of mining tasks, human observation, intervention, and supervision are still integrally involved in these systems (Hughes and Lewis, 2005). The benefits of using teleoperation are already apparent in numerous areas, ranging from space exploration, inspection, robotic navigation, surveillance, underwater operations and rescue activities, etc. Due to the fast improving inexpensive broadband networks, users are able to be situated thousands of kilometers away to accomplish control tasks, allowing them to work under a more convenient condition or to use their skills at many different sites around the world. Particular examples like: web-based robot control (Taylor et al , 1999), 3 internet-based remote control for mining machinery (Kizil and Hancock, 2008), remote surgery and simulated collaborative surgical training using haptic devices (Gunn et al , 2005), etc. Within the teleoperation model, the user interface is always the most fundamental and essential component that affects the entire teleoperation performance significantly. In fact, observations still indicate that directly controlling a robot while watching a video feed from the remote camera(s) remains the most common interaction form in teleoperation (Fong and Thorpe, 2001). Therefore, the basic perceptual link between the user end and the remote environment is generally through a live video stream from a remote camera or a set of cameras as the foundational situational awareness for the operator. The requirements for user interfaces for teleoperation of mining vehicles and systems has been briefly discussed in (Hainsworth, 2001), with the demonstrations of two teleoperated mining systems. In this paper, it clearly indicates that conventional user interfaces such as joysticks, switches, wheels, mice and keyboards are still the major control elements used in mining teleoperation. They are relatively simple, sophisticated, allowing teleoperation to be a viable and profitable technique, which satisfy the basic client requirements for mining systems of robustness and reliability. However, in most teleoperation settings, the operator often has multiple devices to control simultaneously, for example, controlling a robot and a remote camera at the same time. Using conventional interfaces will lead to frequent attention and hands switches between different control tasks and interfaces. It distracts the operator from concentrating on the primary control task, reduces the productivity of the entire process, and increases both workload and the number of avoidable operational mistakes. This paper particularly addresses this attention and hands switch problem in a situation where an operator is controlling one or more remote cameras meanwhile carrying out other mining teleoperation tasks. Instead of using conventional control interfaces to overload the operator’s control ability, we present a novel design that uses human eye gaze as an alternative input for the remote camera control using computer vision based eye-tracking technology. With the user evaluation of a modeled experiment for an implemented prototype system, experimental data was gathered through both objective (performance) measures and subjective (user preference) measures. By the further result analysis, we demonstrate the effectiveness of using the eye-tracking remote camera control for resolving this common problem in mining teleoperation. PROJECT BACKGROUND AND RELATED WORK Remote rock breaking in mining teleoperation The telerobotic rock breaker (Duff et al , 2009) was a recent collaborative project between CSIRO and Rio Tinto Iron Ore. CSIRO was contracted to develop a remote control system to the primary rock breaker at the West Angelas mine (see Figure 1), situated over 1000km north-east of Perth in Western Australia. The rock breaker on the mine site is a serial link manipulator arm with a large hydraulic hammer at the tip to break oversized rocks. The arm is installed at a ROM (Run of Mine) bin, where a number of horizontal bars (referred to as a grizzly) are fitted at the bottom in order to prevent oversized rocks from entering the crusher below. Figure 2 shows the overview of the rock breaker (left) and the grizzly in the ROM bin (right). 4 Fig 1 – West Angelas iron ore mine in Western Australia. Fig 2 – Overview of the rock breaker (left), ROM bin with a grizzly at the bottom (right). On the remote mine site, a number of haul trucks with ore from a nearby quarry will be delivering their load into the bin (see Figure 3). The operator is required to break those oversized rocks stuck on the grizzly by operating a two-handed joystick controller. The operator has limited time to break the rocks, as trucks arrive at short intervals (about 90 seconds). Since dumping a load raises a large could of dust, a water spray is used to settle the dust, which requires about 30 seconds making the operator have a clear vision of the bin. 5 Therefore, the operator only has about 60 seconds to move the arm from its rest position, place it carefully onto a rock, break it by firing the jackhammer, and return the arm to the rest before next truck arrives. The actual remote rock breaking process is shown in Figure 4, which allows the operator to have a desktop based teleoperation environment and live videos as the visual feedback. Fig 3 – Haul truck dumping a load into the ROM bin. When the operator is trying to break a rock, it is necessary for them to have a close view of the target so that detailed information can be obtained to specify the spot on the target for positioning the tip and firing the hammer. It is practically impossible to mount the remote camera on the arm to couple the camera motion to the control of the remote robot like most telerobotic or vehicle settings for reducing the control complexity, as the camera would be easily damaged when the hammer on the tip is being fired to break a rock. The remote camera is actually installed on the side of the bin with a zoomed-in view transferring the live video back to the operator. The operator has to use another joystick controller to control the camera motion for adjusting the view of the target rock in order to complete the breaking spot inspection process then move on to the rock breaker arm control.