sensors Article Web AR Solution for UAV Pilot Training and Usability Testing Roberto Ribeiro 1 , João Ramos 1 , David Safadinho 1 , Arsénio Reis 2 , Carlos Rabadão 1 , João Barroso 2 and António Pereira 1,3,* 1 Computer Science and Communication Research Centre, School of Technology and Management, Polytechnic Institute of Leiria, Campus 2, Morro do Lena-Alto do Vieiro, Apartado 4163, 2411-901 Leiria, Portugal; [email protected] (R.R.); [email protected] (J.R.); [email protected] (D.S.); [email protected] (C.R.) 2 INESC TEC, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal; [email protected] (A.R.); [email protected] (J.B.) 3 INOV INESC Inovação, Institute of New Technologies, Leiria Office, Campus 2, Morro do Lena-Alto do Vieiro, Apartado 4163, 2411-901 Leiria, Portugal * Correspondence: [email protected] Abstract: Data and services are available anywhere at any time thanks to the Internet and mobile devices. Nowadays, there are new ways of representing data through trendy technologies such as augmented reality (AR), which extends our perception of reality through the addition of a virtual layer on top of real-time images. The great potential of unmanned aerial vehicles (UAVs) for carrying out routine and professional tasks has encouraged their use in the creation of several services, such as package delivery or industrial maintenance. Unfortunately, drone piloting is difficult to learn and requires specific training. Since regular training is performed with virtual simulations, we decided to propose a multiplatform cloud-hosted solution based in Web AR for drone training and usability testing. This solution defines a configurable trajectory through virtual elements represented over barcode markers placed on a real environment. The main goal is to provide an inclusive and Citation: Ribeiro, R.; Ramos, J.; accessible training solution which could be used by anyone who wants to learn how to pilot or Safadinho, D.; Reis, A.; Rabadão, C.; Barroso, J.; Pereira, A. Web AR test research related to UAV control. For this paper, we reviewed drones, AR, and human–drone Solution for UAV Pilot Training and interaction (HDI) to propose an architecture and implement a prototype, which was built using a Usability Testing. Sensors 2021, 21, Raspberry Pi 3, a camera, and barcode markers. The validation was conducted using several test 1456. https://doi.org/10.3390/ scenarios. The results show that a real-time AR experience for drone pilot training and usability s21041456 testing is achievable through web technologies. Some of the advantages of this approach, compared to traditional methods, are its high availability by using the web and other ubiquitous devices; the Academic Editor: Enrico Natalizio minimization of technophobia related to crashes; and the development of cost-effective alternatives to train pilots and make the testing phase easier for drone researchers and developers through Received: 10 January 2021 trendy technologies. Accepted: 11 February 2021 Published: 19 February 2021 Keywords: augmented reality; AR obstacle courses; human–drone interaction; marker-based AR; unmanned aerial vehicles; UAV control interfaces; UAV pilot training; Web AR Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affil- iations. 1. Introduction The great advantage of unmanned aerial vehicle (UAV) usage includes the exploration of locations that are difficult to access and data gathering from an aerial perspective, which makes it useful in different areas and scenarios. The countless benefits and potential Copyright: © 2021 by the authors. Licensee MDPI, Basel, Switzerland. of drones are strongly related to their capacity to perform tasks that are difficult, dull, This article is an open access article and dangerous [1], which is being intensively explored either for professional or routine distributed under the terms and activities [2]. Nonetheless, as stated in Cisco’s 2020 Global Networking Trends Report, conditions of the Creative Commons immersive experiences powered with virtual reality (VR) and augmented reality (AR) are Attribution (CC BY) license (https:// trending and emergent technologies [3]. Whilst VR allows the user to interact with an creativecommons.org/licenses/by/ immersive and entirely virtual application, AR can present a layer of virtual content over a 4.0/). layer of real-world images captured in real time. The latter can help users improve their Sensors 2021, 21, 1456. https://doi.org/10.3390/s21041456 https://www.mdpi.com/journal/sensors Sensors 2021, 21, 1456 2 of 32 Sensors 2021, 21, 1456 2 of 30 a layer of real-world images captured in real time. The latter can help users improve their performanceperformance inin many many tasks tasks by by providing providing pertinent pertinent information information about about the picturedthe pictured scenario. sce- Mixednario. Mixed reality reality (MR) extends(MR) extends the concept the concept of AR of with AR featureswith features to detect to detect changes changes in the in real the world,real world, allowing allowing the virtual the virtual elements elements to react to react and and adapt, adapt, e.g., e.g., by simulating by simulating the collisionthe colli- ofsion a virtualof a virtual ball ball with with a real a real wall wall [4]. [4]. VR, VR, AR, AR, and and MR MR are are represented represented by by Milgram Milgram andand KishinoKishino through a virtuality continuum continuum diagram diagram [5], [5], presented presented in in Figure Figure 1.1 .Whilst Whilst VR, VR, at atthe the rightmost rightmost end, end, is represented is represented by bya fully a fully virtual virtual environment, environment, AR AR remains remains closer closer to the to theexperiences experiences comprising comprising a real a real environment. environment. Additionally, Additionally, MR MR brings brings the thetwo two ends ends of the of thecontinuum continuum together, together, i.e., i.e.,reality reality and andvirtuality. virtuality. The Thethree three concepts concepts are grouped are grouped in the in gen- the generaleral concept concept of extended of extended reality reality (XR), (XR), which which brings brings them them all alltogether together [6]. [ 6]. FigureFigure 1.1. AdaptationAdaptation ofof thethe MilgramMilgram andand Kishino’sKishino’s virtualityvirtuality continuumcontinuum [[5].5]. ThisThis studystudy isis motivatedmotivated byby thethe needneed toto simplifysimplify andand improveimprove dronedrone pilotpilot trainingtraining andand usabilityusability testing,testing, thusthus decreasingdecreasing technophobiatechnophobia andand thethe exclusionexclusion thatthat stillstill existexist regardingregarding trendytrendy technologiestechnologies like like UAVs UAVs and and AR. AR. The The great grea majorityt majority of training of training alternatives alternatives for drone for pilotsdrone are pilots performed are performed in entirely in entirely virtual simulators,virtual simulators, which detaches which detaches people frompeople the from feeling the offeeling piloting of piloting a real drone a real withdrone a with real controllera real controller in a real in a environment. real environment. This This motivates motivates the combinationthe combination of UAVs of UAVs and ARand to AR create to create training training experiences experiences that enhance that enhance the real the world real withworld virtual with virtual elements. elements. WeWe identifiedidentified andand addressedaddressed thethefollowing followingissues: issues: • • TheThe controlcontrol of of UAVs UAVs and and their their flight flight dynamics dynamics tend tend to beto difficultbe difficult for newfor new users users to un- to derstand,understand, especially especially for thosefor those who who are not are used not used to traditional to traditional remote remote controllers controllers (RCs). • Traditional AR experiences based on surface detection are limited to enthusiasts, since (RCs). high-end hardware, such as HoloLens or Magic Leap 1, is required to run them. In the • Traditional AR experiences based on surface detection are limited to enthusiasts, case of mobile devices, the oftware Development Kits (SDKs) for AR development are since high-end hardware, such as HoloLens or Magic Leap 1, is required to run them. compatible with a specific list of devices which tend to be expensive, consequently In the case of mobile devices, the oftware Development Kits (SDKs) for AR develop- reducing the number of users that can benefit from the technology in their daily lives. ment are compatible with a specific list of devices which tend to be expensive, con- • For UAV usability testing, developers usually set up simple scenarios and trajectories sequently reducing the number of users that can benefit from the technology in their made of real obstacles. Every time a new trajectory is needed, the real obstacles (e.g., daily lives. cones, poles, loops) must be moved. Though higher obstacles bring added value • For UAV usability testing, developers usually set up simple scenarios and trajectories to tests of aerial maneuvers, it can be difficult and dangerous to change an obstacle made of real obstacles. Every time a new trajectory is needed, the real obstacles (e.g., every time a new trajectory is required, for instance, when it is placed on the top of a cones, poles, loops) must be moved. Though higher obstacles bring added value to high wall. • Newtests usersof aerial are maneuvers,