Predicting Rifle Shooting Accuracy from Context and Sensor Data
Total Page:16
File Type:pdf, Size:1020Kb
Predicting rifle shooting accuracy from context and sensor data A study of how to perform data mining and knowledge discovery in the target shooting domain PAPER WITHIN: Computer Science AUTHORS: Viktor Jansson, Max Pettersson TUTOR: Niklas Lavesson JÖNKÖPING 2019 July This exam work has been carried out at the School of Engineering in Jönköping in the subject area computer science. The work is a part of the three-year Bachelor of Science in Engineering programme. The authors take full responsibility for opinions, conclusions and findings presented. Examiner: Tuwe Löfström Supervisor: Niklas Lavesson Scope: 15 hp Date: 2019-07-24 Mailing address: Visiting address: Phone: Box 1026 Gjuterigatan 5 036-10 10 00 (vx) 551 11 Jönköping PREDICTING RIFLE SHOOTING ACCURACY I Abstract The purpose of this thesis is to develop an interpretable model that gives predictions for what factors impacted a shooter’s results. Experiment is our chosen research method. Our three independent variables are weapon movement, trigger pull force and heart rate. Our dependent variable is shooting accuracy. A random forest regression model is trained with the experiment data to produce predictions of shooting accuracy and to show correlation between independent and dependent variables. Our method shows that an increase in weapon movement, trigger pull force and heart rate decrease the predicted accuracy score. Weapon movement impacted shooting results the most with 53.61%, while trigger pull force and heart rate impacted shooting results 22.20% and 24.18% respectively. We have also shown that LIME can be a viable method to give explanations on how the measured factors impacted shooting results. The results from this thesis lay the groundwork for better training tools for target shooting using explainable prediction models with sensors. Keywords: Interpretability - Target shooting - Regression trees - Feature selection - Cross-validation PREDICTING RIFLE SHOOTING ACCURACY II Acknowledgments We would like to thank Niklas Lavesson for the guidance we received on our thesis. And for the opportunity to do our thesis within MAPPE with support from the Knowledge foundation. We would also like to thank Andreas Månsson from SAAB AB Training & simulations (SAAB) for helping us with equipment and contact with SAAB. We also thank Anton Berkman and Gustav Andersson who helped us with our experiment. PREDICTING RIFLE SHOOTING ACCURACY III Contents 1 Introduction 1 1.1 Background . .2 1.2 Problem description . .5 1.3 Research question . .6 2 Method 7 2.1 Experiment design . .7 2.2 Independent variables . .9 2.2.1 Heart rate . .9 2.2.2 Trigger pull force . .9 2.2.3 Rifle movement . 11 2.3 Dependent variable . 12 2.3.1 Target hits . 12 2.4 Experiment execution . 14 2.5 Data processing . 15 2.6 Random forest regression . 16 3 Results 17 3.1 Analysis . 20 4 Discussion 22 4.1 Results . 22 PREDICTING RIFLE SHOOTING ACCURACY IV 4.2 Validity and limitations . 23 4.3 Future work . 24 4.4 Conclusions . 25 References 26 A Appendix 29 B Appendix 30 C Appendix 32 PREDICTING RIFLE SHOOTING ACCURACY 1 1 Introduction The aim of this thesis is to further study how to perform data mining and knowledge discovery in the target shooting domain with the goal to generate interpretable models or model explanations. This thesis specifically tries to determine a method to give predictions for what factors impacts a shooter’s results. The independent variables are rifle movement, trigger pull force and heart rate. These variables will be studied to determine the correlation with the dependent variable shooting accuracy. Target shooting is a broad activity that has use in sports shooting and weapons training both in law enforcement and in the military sector. Multiple studies have been conducted on target shooting that shows that it is a good avenue to apply statistical prediction models to find ways to improve shooting accuracy (i.e Deng, Liu, and Hsieh, 2011; Lin and Wu, 2012 and Maier, Meister, Trösch, and Wehrlin, 2018). The use of machine learning could potentially introduce new ways to conduct training within target shooting. This could be beneficial to a large audience of sports shooting practitioners and hunters as well as police and military personnel. Machine learning is the study of statistical models and algorithms within the context of computer systems. One of the tools of machine learning are learning algorithms that take data and trains a predictive model that can then predict future outcomes. It is a fast-growing field and learning algorithms can outperform humans in specific areas (Doshi-Velez & Kim, 2017). It has been successfully used to analyze, for example, performance in sports (Novatchkov & Baca, 2013). However, a problem with this growth in popularity is algorithms where the user does not know why the algorithm makes the prediction it does. According to Doshi-Velez and Kim (2017) is it important for a machine learning model to be explained in cases where you might want a deeper knowledge of the problem that the model is trying to solve. This thesis is part of a larger research project called MAPPE (Mining Actionable Patterns from complex Physical Environments) which explores how to design and evaluate predictive models that can provide explanations for their predictions. MAPPE is grounded in the model LIME (Local Interpretable Model-agnostic Explanations). The goal of this thesis is partly in support of the goals of that PREDICTING RIFLE SHOOTING ACCURACY 2 project. 1.1 Background A black-box algorithm is an algorithm where the inner workings are unknown to the user, which means that the user does not know how the algorithm made its prediction (Guidotti et al., 2018). For some areas, this is not an issue, e.g. ad servers. However, as machine learning grows more popular and is used in more areas, factors other than task performance become more important, such as security and nondiscrimination (Doshi-Velez & Kim, 2017). This means that underlying structural or societal patterns can incorrectly train a predictive model, resulting in incorrect predictions and even discrimination against groups of people. Guidotti et al. (2018) gives an example of such a problem where a military application of image recognition of tanks seemingly made correct predictions of enemy versus friendly tanks. It turns out, however, that the algorithm was actually trained to look at whether the picture was taken on a clear or overcast day, not the actual features of the tanks. This problem of interpretability of algorithms has been recognized by the European union with the new legislation that essentially creates a "right to explanation" rule, where an algorithmic decision needs to be able to provide an explanation for how it came to a decision (Goodman & Flaxman, 2017). Interpretability within machine learning, or machine learning explainers, is a fairly new sub-field originating from the need for interpretable algorithms (Gilpin et al., 2018). To address the question of interpretability within machine learning several methods have recently been developed, Gilpin et al. (2018) and Guidotti et al. (2018) lists and categorize some of these. One recent example is LIME, which was developed to interpret and give explanations for any machine learning algorithm (Ribeiro, Singh, & Guestrin, 2016). LIME is categorized as a linear proxy model. This means that LIME will slightly change the inputs for a black-box model and see what happens with the output. From this relationship between in- and output, LIME can then construct its own proxy model for the black-box within the scope of the inputs. This means that LIME is locally interpretable, i.e. it creates a proxy model for the instance of the inputs, but not for the entire black-box model. This method also makes it model-agnostic. This means that LIME does not have to know anything about the inner workings of the black-box model since it only PREDICTING RIFLE SHOOTING ACCURACY 3 looks at in- and outputs. The result is that LIME can show which of the inputs had the largest impact in how the black-box model made its prediction (Ribeiro et al., 2016). LIME has been used in different areas of research, for example to aid with human decision making for a model trained on radar signal recognition (W. Zhang, Ge, Jin, & Guo, 2018) and in an expanded version for the use in music content analysis (Mishra, Sturm, & Dixon, 2017). Because LIME is a relatively new method, there are several areas in which it has not yet been applied, for instance in target shooting. There are many factors to determine shooting accuracy to explain certain shooting results. Ball, Best, and Wrigley (2003) looked at how body sway and aim point fluctuation impacted performance. Era, Konttinen, Mehto, Saarela, and Lyytinen (1996) studied how posture control between experienced and inexperienced shooters impacted performance. Z. S. Zhang, Qiu, Liu, Huang, and Wang (2013) and Lin and Wu (2012) looked at aiming accuracy, hitting stability and trigger control. In a handbook about pistol marksmanship (USMC, 2001) it is stated that aiming, trigger control and breath control are the most important factors. All these studies point to weapon movement and weapon control to be the main factors in determining shooting results and this will form the basis of what this thesis looks at. There are numerous studies which have used statistical and learning algorithms to draw conclusions on shooting accuracy. Deng et al. (2011) used three different prediction models to look at how a shooter grips a pistol and pulls the trigger. In addition to concluding that weapon stability and correct trigger pull are the main factors for shooting accuracy. Lin and Wu (2012) studied how to design an intelligent evaluation system for shooting, by measuring factors of distance of the hit to the bull’s eye, weapon holding stability and shaking intensity during the shot.