Practical Evaluation Framework for PDR Compared to Reference Localization Methods
Total Page:16
File Type:pdf, Size:1020Kb
2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan Practical Evaluation Framework for PDR Compared to Reference Localization Methods Ryosuke Ichikari, Ching-Tzun Chang, Masakatsu Kourogi, Takashi Okuma, and Takeshi Kurata Human Informatics Research Institute AIST Tsukuba, Ibaraki, Japan Abstract— Pedestrian dead-reckoning (PDR) yields practical sharing dataset and comparing with ground truth data [1][2]. applications; thus, evaluations based on objective criteria This type can be categorized as offline benchmarking. Another between such technologies are important for both researchers type of evaluation is on-site bench marking or on-site and application developers. One of the most important elements competition. In augmented reality (AR) / mixed reality (MR) of objective evaluations of PDR is ground truth reference data. research area, there is a tracking competition [3] for AR / MR To evaluate PDR methods in practical contexts, e.g., in public spaces and factory environments, the reference data should be camera tracking methods in controlled indoor environments. prepared in the same environments in a practical manner in The indoor localization research community also has on-site acceptable time with a manageable amount of intervention. For localization competitions [4-7]. PDR Challenge [6] was held performance improvement by trial and error, developers and as a competition focused on PDR. As a derivative off-site researchers would benefit from the availability of sufficiently rich competition of the PDR challenge, the PDR bench marking data to reconstruct and analyze a target’ situation. Therefore, we standardization committee in Japan will hold “PDR challenge propose a practical framework to evaluate PDR methods. We in warehouse picking” [8] in which for PDR methods will have utilized additional localization methods such as Google’s compete in the warehouse context at International Conference Tango, Noitom’s Perception Neuron, and Microsoft’s HoloLens on Indoor Positioning and Indoor Navigation (IPIN 2017). as practical ways to prepare reference data in various environments to evaluate PDR methods and reconstruct target One of the most important elements for objective evaluation situations. In addition, we have developed a visualization tool to of PDR is the preparation of accurate reference data to be used play-back estimated trajectories while richly reconstructing a as ground truth data. To evaluate PDR methods in practical target’s walking situations using the reference data. The contexts, such as public spaces and factory environments, the effectiveness of the proposed framework is demonstrated by reference data should be prepared in the same environment preparing test dataset with realistic scenario for PDR and using a practical approach that can be completed in acceptable comparing PDR methods to the dataset. time with an appropriate amount of intervention. With respect to performance improvement, developers and researchers Keywords—pedestrian dead-reckoning; benchmarking; motion capture; objective evaluation; would greatly appreciate high-availability rich reference data. In this paper, we propose a practical framework to evaluate PDR methods. The proposed evaluation framework has I. BENCHMARKING FOR PEDESTRIAN DEAD-RECKONING following the characteristics. Pedestrian dead-reckoning (PDR) yields practical - Enables simple preparation of reference data. applications; thus, evaluations based on objective criteria - The reference data include the position and orientation between technologies are become important for researchers of the targets, as well as additional information for and application developers. PDR has a variety of reconstructing target situations such as whole body methodologies and elemental technologies such as foot- posture and how to carry the device. mounted zero-velocity update, walking pattern recognition - An intuitive visualization tool is provided to compare with an accelerometer, and moving direction estimation. To various types of methods with a reconstructed target allow a service provider to select an appropriate method to situation with synchronization in 3D space. start a service with PDR, it is important to consolidate The remainder of this paper describes the requirements and frameworks to fairly compare the performance of various PDR concepts of the proposed evaluation frameworks and methods, as well as the advantages and disadvantages of such introduces an actual test dataset and evaluation example with methods. PDR researchers and developers also require the dataset embodied concepts. objective criteria and test beds to improve the performance of their methods. II. EVALUATION FRAMEWORK REQUIREMENTS In computer vision and pattern recognition fields, bench Here, we list the requirements of the PDR evaluation. marking is commonly employed to evaluate methods by - To allow many potential PDR developers to utilize the © IPIN2017 dataset, it should contain various types of high- 2017 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 18-21 September 2017, Sapporo, Japan technology enables very accurate camera tracking by utilizing a built-in wide-angle camera and a depth camera. We utilized the Tango SDK’s “Motion Tracking” camera HoloLens tracking method to obtain highly accurate position and Perception orientation. We found that this functionality was accurate Neuron Smartphone and robust, even though the Tango device moves quickly (PDR) when the user waves their arm. - Microsoft HoloLens Tango The HoloLens is an MR (similar concepts to AR) smartglass device provided by Microsoft. The HoloLens has wide-angle cameras and depth cameras as well as optical see-through displays. This device enables highly accurate camera tracking when reconstructing a surrounding Figure 1: Data capture configuration environment. Even though the HoloLens has MR functionality, we utilized its camera tracking function to frequency raw sensor data, such as angular velocity, obtain absolute position and posture information as acceleration, earth magnetic measurements, and reference data. atmospheric pressure. - Noitom’s Perception Neuron - The dataset should contain additional reference data The Perception Neuron (PN) is an inertial measurement that can be used to reconstruct a highly detailed target unit (IMU)-based motion capture technology developed by situation. Noitom. Common motion capture systems, such as Vicon - The dataset should contain positional reference data and MAC3D, estimate target’s whole body posture using that can be regarded as ground truth data. retro-reflective markers and infrared cameras arranged in - Positional reference data should be measured by studio environments; thus, the coverage area of such system minimum effort. is limited by the camera arrangement. In contrast, the - The dataset should include realistic data captured in coverage area of the PN is not limited by camera realistic scenarios. arrangement. As a result, the PN can easily begin motion - The content of the dataset should be synchronized capture wherever the user desires by simply wearing the using a shared time code. sensor suit and measuring simple calibration poses. In this - The dataset should be provided along with a measurement, we utilized whole body posture estimated visualization tool that can intuitively compare using the PN to reconstruct a target situation. The PN can estimated trajectories, the reference data, and the also estimate the relative movement of the target by reconstructed target situation. utilizing the whole body posture and ground contact estimation. III. DATASET PREPARATION - Video recording We also recorded video when measuring the data. The A. Device configuration for measuring dataset recorded video was utilized to confirm the measuring As an example of a dataset that fulfills the above situation. This can be visualized using a visualization tool requirements, we conducted experimental measurements by and other reference data. preparing a dataset in an office environment at our facility. We B. Scenarios for the dataset utilized various localization technologies with different characteristics that were measured simultaneously by We conducted measurement of the dataset under different employing time synchronization. The measurement scenarios and device configurations, as shown in Table1. configuration is shown in Figure 1, and the concrete Detailed descriptions of each scenario are described as follows. configuration of the measurement process comprises the Note that the walking paths were always the same during the following elements. measurement as, shown in Figure 2. - Raw sensor data logging by a smartphone for PDR We captured raw sensor data using a Nexus 5 Android Table1: List of sample datasets No. PDR measurement style Tango HoloLens PN smartphone. The sensor data included high-frequency 1 Right Hand (Navigation) Right Hand (Navigation) - ✔ (Whole body with 18NEURONs) angular velocity, acceleration, earth magnetic measurements, 2 Fixed on waist Fixed on waist - ✔ and atmospheric pressure measurements. The positions of 3 Fixed on waist Fixed on waist - ✔ 4 Mounted in chest pocket Fixed on waist ✔ ✔ the device on the target’s body were varied according to 5 Mounted in chest pocket Fixed on waist ✔ ✔ various scenarios. 6 Right Hand (Navigation) Right Hand (Navigation) ✔ ✔ 7 Right Hand (Navigation) Right Hand (Navigation) - ✔ - Google Tango 8 Right Hand (Arm-waving) Right Hand (Arm-waving)