Edinburgh Research Explorer Calibrating Recurrent Neural Networks on Smartphone Inertial Sensors for Location Tracking Citation for published version: Wei, X & Radu, V 2019, Calibrating Recurrent Neural Networks on Smartphone Inertial Sensors for Location Tracking. in 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN). Institute of Electrical and Electronics Engineers (IEEE), pp. 1-8, 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN), Pisa, Italy, 30/09/19. https://doi.org/10.1109/IPIN.2019.8911768 Digital Object Identifier (DOI): 10.1109/IPIN.2019.8911768 Link: Link to publication record in Edinburgh Research Explorer Document Version: Early version, also known as pre-print Published In: 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN) General rights Copyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s) and / or other copyright owners and it is a condition of accessing these publications that users recognise and abide by the legal requirements associated with these rights. Take down policy The University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorer content complies with UK legislation. If you believe that the public display of this file breaches copyright please contact [email protected] providing details, and we will remove access to the work immediately and investigate your claim. Download date: 07. Oct. 2021 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 30 Sept. - 3 Oct. 2019, Pisa, Italy Calibrating Recurrent Neural Networks on Smartphone Inertial Sensors for Location Tracking Xijia Wei Valentin Radu University of Edinburgh University of Edinburgh [email protected] [email protected] Abstract—The need for location tracking in many mobile skills. The problem with this manually engineered approach services has given rise to the broad research topic of indoor is that edge cases will always exist that are hard to formulate positioning we see today. However, the majority of proposed and integrate in these systems, which is also the reason for systems in this space is based on traditional approaches of signal processing and simple machine learning solutions. In the wide performance variation we see between such systems. the age of big data, it is imperative to evolve our techniques We argue that manual formulation of the location estimation to learn the complexity of indoor environments directly from process is limited and so we should rely on automatic learning data with modern machine learning approaches inspired from directly from data instead, without much human intervention. deep learning. We model location tracking from smartphone In this age of data driven systems, adopting modern machine inertial sensor data with recurrent neural networks. Through our broad experimentation we provide an empirical study of the best learning techniques, such a deep learning, will help to move model configuration, data preprocessing and training process to our community forward. achieve improved inference accuracy. Our explored solutions are In this work we explore a robust modelling solution, Recur- lightweight to run efficiently under limited computing resources available on mobile devices, while also achieving accurate es- rent Neural Networks (RNN) for the task of position tracking timations, within 5 meters median error from inertial sensors on smartphone inertial sensors. RNNs have proven effective alone. in other sequence based tasks, such as in machine translation, Index Terms—deep learning, location tracking, indoor localiza- speech, and natural language processing. We explore a range tion, inertial sensors, recurrent neural networks, dead reckoning of data preprocessing choices and model configurations to assess their impact on location estimation accuracy by training I. INTRODUCTION several different models. We find that data down-sampling is A growing number of location based services has given beneficial to having a smaller model that can run on mobile rise to the research topic of position estimation, proposing devices, while achieving below 5 meters median error, and innovative solutions for the more difficult cases, such as for time window overlapping helps to strengthen observations in indoors where access to GPS is unreliable. Using the inertial the model while also expanding the available training data to sensors available on smartphones (accelerometer, gyroscope benefit training. Transferring models training on data from one and magnetometer) good position estimations are achievable, device to perform estimation on another device is also explored although not without limitations. here, showing the good generalization of RNN models. Inertial sensors are commonly used to construct Dead Although we move the burden of developing localization Reckoning systems, taking a confident observation as starting systems to generating good labeled training sets, we believe point, followed by consecutive location estimations on top this is more scalable since data collection is easier than human of previous locations by determining direction of movement intervention to alter previous systems for new environments and traveled distance [1]. However, a severe problem with and edge cases. Solutions based on infrastructure cameras to this approach is that occasional erroneous estimations (due extract location estimation [7] for sensor data labeling can be to sensor noise, drift and device calibration) are cumulative one approach to enhance training data collection at scale. in the system, leading to increasing estimation errors very This paper makes the following contributions: fast [2]. For this reason, Dead Reckoning is often augmented with opportunistic anchoring to the physical space, either by • We formulate location tracking as a recurrent neural identifying unique signatures of sensors [3], activity recogni- network problem, incorporating all the complexity of tion [4], ambient conditions [5], collaborative estimation [1] mobility model generation into an automatic learning or by radio signal signatures [6]. process from location labeled sensor data. Exploring the literature, the vast majority of previous so- • Training and testing of recurrent neural networks is done lutions to perform location estimation from inertial sensors on a sizable dataset we collect for this exploration. proposes heavily engineered approaches. These are as good as • We offer insights into the best options to calibrate recur- the quality of human expert observations and their modelling rent neural networks to achieve improvements in location 978-1-7281-1788-1/19/$31.00 © 2019 IEEE estimation with these recurrent neural network models. 2019 International Conference on Indoor Positioning and Indoor Navigation (IPIN), 30 Sept. - 3 Oct. 2019, Pisa, Italy ht-1 ht ht+1 ht+2 term memory cannot accumulate indefinitely, so a forget gate is used to keep the size tractable. Figure 2 shows the internal Ct-1, ht-1 Ct, ht Ct+1, ht+1 C h C h C h C h t-1 t-1 t t t+1 t+1 t+2 t+2 structure of one LSTM unit. In each unit, there are not only input and output gates but also a forget gate that controls x x x x t-1 t t+1 t+2 the amount of information propagated to the next block and what is dropped in the current stage [10]. The input to a Fig. 1. The structure of a recurrent neural network with LSTM units, which estimates an output ht based on an input Xt and information received from block for us is a concatenation of acceleration, gyroscope and previous blocks in the chain (Ct−1 and ht−1). magnetometer values over a time window. The value in the current state is controlled by the forget h t gate f signal. Specifically, this saves the value when the signal is set to 1 and forgets when the gate is set to 0. The + Ct-1 Ct tanh activation of receiving a new input and propagating this are ft it ot C h σ σ σ C h determined by signals to the input gate and to the output t-1 t-1 tanh t+1 t+1 h t gate respectively [11]. Equations 1 to 6 show the formulation + + + + h t-1 of transformations performed inside the block, where W are weights learnt in training. xt i = σ(W x + W m ) (1) Fig. 2. The flow in one LSTM unit, showing long-term memory accumulation t ix t im t−1 in Ct and short-term term memory representing the output of previous unit ft = σ(Wfxxt + Wfmmt−1) (2) ht−1. ot = σ(Woxxt + Wommt−1) (3) ct = ft ct−1 + it h(Wcxxt + Wcmmt−1) (4) II. METHODS mt = ot ct (5) This section presents the deep learning technique we adopt pt+1 = Softmax(mt) (6) to perform indoor localization on smartphone inertial sensor data. We adopt a validated recurrent neural network technique As inertial sensor data is streamed in time sequences, the showing good performance in other domains to produce a LSTM model is ideal for location estimations on this type of modern perspective to the classic dead reckoning solution. data. The size of one sample is time step ∗ no features, where features are the magnitude of measured values on A. Dead Reckoning as Recurrent Neural Network the three axes of each sensor, accelerometer, gyroscope and Dead Reckoning is the process of estimating continuous magnetic sensor. A time window is formed of a number of locations by starting from a known point (e.g., by detecting sensor signal samples collected over such interval of time and entrances [8]) and estimating consecutive positions based on regularised to a fixed sampling rate. Each data instance has a a stream of observations (direction of movement and dis- target position (Xi, Yi) as label. On this data representation, placement). This resembles the process performed by recurrent the LSTM model produces position estimations in coordinates neural networks, building on previous estimations (or features (Xest, Yest). This is formulated by equations 7–9. from previous estimations) and on new environment obser- vations to produce a sequence of predictions.