DJI Government Edition UAS Data Management Assurance Flight Test and Evaluation Report

Flight Test and Technical Evaluation Report DJI Unmanned Aircraft System (UAS) Mission Functionality and Data Management Assurance Assessment U.S. Department of the Interior Office of Aviation Services Boise, Idaho by Mark L Bathrick Brad Koeckeritz July 2, 2019 This Flight Test and Technical Evaluation Report presents results of an assessment of the flight, payload, and data management assurance performance of the SZ DJI Technology Co., Ltd. Matrice 600 Pro and Mavic Pro unmanned aircraft systems (UAS) modified with DJI Government Edition (GE) hardware, firmware, and software for suitability in selected Interior bureau missions. Test articles were operated under the Office of Aviation Services (OAS) Test Plan for Validation and Verification of Data Security Software for Employment of DJI UAS Platforms, originally signed on April 9, 2018 and later updated with approved Modifications One and Two of November 14 and 16, 2018, respectively. APPROVED FOR RELEASE: Mark L Bathrick, Director Page 1 of 53 SUMMARY In 2014, following four years of unmanned aircraft systems (UAS) operational test and evaluation (OT&E), the U.S. Department of the Interior’s (DOI) Office of Aviation Services (OAS) and over 300 subject matter experts from DOI bureaus developed the Master UAS Requirements for the DOI. In 2015, while conducting UAS market research, OAS determined the privacy policy of the largest provider of UAS in the U.S. (SZ DJI Technology Co., Ltd) did not meet UAS data management assurance standards contained in the Master UAS Requirements for the DOI. Specifically, they did not meet DOI’s requirement to be able “to decline and lock out any device information sharing including telemetry through aircraft, software or applications preventing any automated uploads or downloads.” In 2016, Interior awarded its first contract for fleet small UAS to a U.S. based company whose aircraft met all relevant technical and DOI bureau price requirements. Shortly thereafter, the company (3D Robotics) ceased manufacturing of UAS as a result of market competition. Subsequent OAS market research to identify additional UAS to meet Interior bureaus’ growing demand for inexpensive and highly capable aircraft indicated the remaining UAS available from U.S. based companies were up to 10X less capable for the same price, or up to 10X more costly than similarly capable DJI aircraft. OAS immediately began working with Interior and federal partners and the drone industry to identify, develop, and field potential solutions that met Interior’s three data management and risk mitigation requirements listed in the Master UAS Requirements for the DOI. In 2017, working with the Department of Defense (DOD), OAS identified a potential solution and began developing a flight test plan to assess the suitability of this solution across a range of DOI bureau UAS missions. In 2017, OAS was also approached by DJI with an offer to collaborate on the development, testing and potential fielding of a customer-focused enterprise solution that would meet Interior’s UAS data management and risk mitigation requirements for enterprise level managed data sharing controls. Draft specifications for the new “Private Edition” (later referred to as “Government Edition” - GE) included custom software, firmware, and UAS hardware editions for two specific Interior selected DJI drones (Matrice 600 Pro, Mavic Pro). Provisions to include flight testing of GE equipped DJI aircraft were later added to the flight test plan. Additionally, OAS collaborated with one industry and two federal partners with expertise in data management assurance testing to conduct targeted assessments of GE hardware, firmware, and software. While the DOD solution was secure, it provided insufficient mission functionality for DOI bureaus’ complex photogrammetry, mapping, etc. mission data requirements. Flight testing of earlier versions of GE identified functional issues that were remedied in later versions. Partner led data management assurance tests of each GE version met required security exit criteria. It is recommended GE (Pilot App version 1.3 19743, Assistant 2 GE Version 9-5) equipped Matrice 600 Pro and Mavic Pro aircraft be authorized for Interior fleet and contract use in accordance with additional risk mitigation practices developed by OAS (Appendices A, B). While the tested GE version met Interior requirements, the necessity to test and validate future GE updates to ensure continued security makes this solution time- consuming and costly to maintain and scale; not a suitable long term solution. Continued collaboration with federal and industry partners to identify additional solutions that meet DOI data management assurance requirements and are easier and less costly to sustain and scale is also recommended. Secure Agile Scalable Page 2 of 53 TABLE OF CONTENTS BACKGROUND 4 UAS DATA MANAGEMENT AND RISK MANAGEMENT REQUIREMENTS 6 DOI UAS DATA MANAGEMENT RISK MITIGATION STRATEGY 6 PURPOSE 9 TEST EQUIPMENT- HARDWARE, FIRMWARE, SOFTWARE AND APPLICATIONS 9 SCOPE OF TESTS 10 TEST OBJECTIVES 10 TEST PHASES AND EXIT CRITERIA STANDARDS 11 FUNCTIONAL AND MISSION FLIGHT TESTING PHASES 11 DATA MANAGEMENT ASSURANCE TESTS AND EXIT CRITERIA 15 ADDITIONAL DATA MANAGEMENT ASSURANCE RISK MITIGATION MEASURES EMPLOYED 17 TEST ENVELOPE 17 TEST CONFIGURATIONS AND LOADING 18 METHOD OF TESTS 18 TEST METHODS AND PROCEDURES 18 CHRONOLOGY 19 RESULTS AND DISCUSSION 19 TEST RESULTS - AS RELATED TO THE OBJECTIVES AND THE MISSION 19 CONCLUSIONS 21 RECOMMENDATIONS 22 ABOUT THE AUTHORS 23 REFERENCES 24 APPENDICES 24 Note: For readers viewing this report electronically, embedded hyperlinks found throughout the report provide access to additional background and reference information. Page 3 of 53 INTRODUCTION BACKGROUND The U.S. Department of the Interior (DOI) is the largest land steward in the United States, responsible for management of 500M acres (~1 in 5) across the U.S and its territories and 1.7B acres on the Outer Continental Shelf. DOI manages resources that supply 30 percent of the Nation’s energy supplies, manages water in 17 Western States and supplies 15 percent of the Nation’s hydropower energy. The “people’s land,” which DOI manages on behalf of the American Taxpayer annually hosts more than 450M visits annually. In carrying out its extensive responsibilities on behalf of the American Public, DOI utilizes a wide variety of aircraft, including unmanned aircraft systems (UAS, aka drones). DOI missions, often conducted in remote areas, severe terrain, and weather conditions can be hazardous to personnel. These missions often require persistent presence and responsive deployment to address emergent events (e.g. wildfires, earthquakes, volcanos, floods, animal migrations, search and rescues, etc.). Mission goals include conducting them with no/minimal disturbance to native species and visitors to the lands that DOI stewards, while making the best use of appropriated funds to fulfill its chartered obligations for managing the “people’s land.” Since the initiation of DOI’s current UAS program in 2006, the Department has realized significant benefits from the safe and responsible integration of drone technology. DOI is a recognized leader in the non-military government use of unmanned aircraft systems for a variety of missions with over 600 available drones and 400 FAA certified and DOI trained operators distributed across 42 states and territories assigned to seven of Interior’s nine bureaus. Since 2010 when Interior began employing UAS in operational missions, the Department has successfully flown over 19,000 UAS flights, with zero public complaints. In 2018, the DOI UAS program continued its tradition of innovation, collaboration, and leadership in the drone space. Adoption and integration of UAS in missions by DOI’s nine bureaus continued to grow with 10,342 UAS flights conducted across more than 25 mission applications in 42 States and U.S. Territories in 2018; a 108% increase in DOI UAS flights over 2017’s record setting year. This included 86 flights and 382 flight hours flown in six states, supporting 16 wildfires, by commercial companies on DOI’s first UAS services call-when-needed contract. Page 4 of 53 Figure 1 – DOI Annual UAS Flight Activity Interior is also recognized as the government leader in cost-effective development and management of its unmanned aircraft systems program. Notable accomplishments in this area include: 1. Interior’s Office of Aviation Services (OAS) developed and oversees Interior’s award-winning UAS program with no additional personnel or funding. 2. All DOI’s approximately 400 FAA certified and DOI trained UAS operators came from current employees; no new hires. 3. DOI leveraged approximately $25M in excess Department of Defense small UAS during a four year operational test and evaluation (OT&E) program that was used to inform the development of DOI’s Master UAS Requirements document. Interior is the only federal agency with published UAS requirements available online to the public. 4. DOI’s entire fleet of >600 fleet UAS cost less than several of the single aircraft in Interior’s 93 aircraft manned fleet, averaging about $3,600 per UAS. 5. In 2018, DOI UAS use on 543 unique projects saved an estimated $14.8M over the cost of traditional ground based methods of accomplishing these projects. Finally, Interior is also a recognized leader in UAS program transparency, privacy, and data security. The Interior Office of Aviation Services public UAS webpages contain over 200 separate technical reports, presentations,

Load more