ICES-2020-428

Astronaut Smart Glove: A Human-Machine Interface For the Exploration of the , and Beyond

Pascal Lee1 Mars Institute, SETI Institute and NASA Ames Research Center, Moffett Field, CA 94035-1000, USA

Christopher P. McKay2 NASA Ames Research Center, Moffett Field, CA 94035-1000, USA

Gregory Quinn3, Thomas Chase4, Jake Rohrig5 Collins Aerospace, 1 Hamilton Road, Windsor Locks, CT 06096, USA

Moina Tamuly6, Sondre Tagestad7, Haakon Pettersen8, Magnus Arveng9, Frank Oygard10 Ntention AS, Granveien 53, 1394, Nesbru, Norway

Brandon Dotson11 Aviation Flight Test Directorate, Redstone Arsenal, AL 35808, USA

and

John W. Schutt12 Mars Institute and SETI Institute, 189 N. Bernardo Ave, Mountain View, CA 94043, USA

Astronauts exploring the Moon, Mars and beyond will be assisted by robotic systems to render their work more efficient, productive, and safe. Among these, unmanned aerial vehicles (UAVs) or drones (airplanes, rotorcraft, or gas thrustered flyers), hold great promise, as they may assist astronauts in a wide range of science and exploration activities. UAV operations, however, are presently demanding tasks. Conventional drone interfaces require significant dexterity and situational awareness to enable subtle and rapid real-time control inputs. Such interfacing would be inadequate if the drone operator were wearing a pressurized spacesuit, as the latter fundamentally limits an astronaut’s ability to perceive and interact with the extra-vehicular environment. During the 2019 campaign of the NASA Haughton-Mars Project (HMP) on , High Arctic, an established Moon and Mars analog field research site, a novel concept for a wireless human-machine interface (HMI) called “Astronaut Smart Glove”(ASG) was field- tested in partially simulated, unpressurized astronaut extra-vehicular activity (EVA). The ASG, along with its compact in-suit augmented reality (AR) head-mounted display (HMD),

1 Director, Mars Institute, NASA Ames Research Center, MS 245-3. E-mail: [email protected] 2 Senior Research Scientist, Space Sciences Division, NASA Ames Research Center, MS 245-3. 3 Principal Research Engineer, Collins Aerospace, Civil Space & Sea Systems, MS 1A-2-W66. 4 Engineer, Collins Aerospace, Civil Space & Sea Systems, MS 1A-2-W66. 5 Engineer, Collins Aerospace, Civil Space & Sea Systems, MS 1A-2-W66. 6 Co-Founder & Co-CEO, Ntention AS. 7 Lead Engineer, Hardware & Software Systems, Ntention AS. 8 CTO, Ntention AS. 9 Co-Founder & Co-CEO, Ntention AS. 10 Officer, Strategy & Research, Ntention AS. 11 Test pilot, Aviation Flight Test Directorate, Redstone Arsenal. 12 Base Manager, Field Operations, Mars Institute. were evaluated for their potential adequacy in allowing UAVs to be operated by a suited astronaut. The ASG showed promise in being able to address both the dexterity and situational awareness limitations of spacesuits by allowing an astronaut to operate single- handedly, within conservative work envelopes for EVA hand operations, a UAV via low amplitude, intuitive gestures of one hand, and in head-up mode via direct visual contact with the UAV and/or in First Person View (FPV) using the AR display. While the ASG offers the prospect of enabling a wide range of robotic operations in future human exploration, further studies are needed to understand better the system’s potential limitations, in particular higher fidelity tests using a pressurized suit, and field demonstrations of end-to-end EVA surface science and exploration operations.

I. Introduction HE Astronaut Smart Glove (ASG) is a concept human-machine interface (HMI) for EVA spacesuits that would T allow astronauts to operate a wide range of robotic systems on the Moon, Mars, and elsewhere in space by simple, intuitive, single-handed hand, wrist, and finger gestures. The ASG is the result of a collaboration between non-profit space research organizations (Mars Institute and SETI Institute in the USA), government (NASA, specifically NASA Ames Research Center), and private industry (Collins Aerospace in the USA and Ntention in Norway). A first prototype of the ASG was designed and developed in Spring 2019, and field-tested as part of a concept spacesuit for future Moon and Mars exploration at the NASA Haughton-Mars Project (HMP) planetary analog field research site on Devon Island, High Arctic, in Summer 2019. This paper presents the context and results of this initial field study. The reported study was motivated, and rendered possible, by the convergence of several key factors: 1) long- standing and ongoing field studies of spacesuit systems at the NASA HMP with Collins Aerospace to develop advanced EVA concepts for Moon and Mars science and exploration, in particular to make future EVAs easier, more productive, more cost-effective, and safer; 2) the ongoing use of drones/UAVs at the NASA HMP in support of field geology and planetary analog field science and exploration studies, and 3) the development by Ntention of a ground- breaking smart glove allowing intuitive, single-handed commercial drone operations. In this paper, we describe the NASA HMP as context for our field study, the need and challenge of creating practical HMIs allowing astronauts on EVA on the Moon, Mars and beyond to interface with a wide range of robotic systems, and the growing importance and promise of UAVs in planetary science and exploration. We then present the ASG concept and design, describe the HMP-2019 ASG field experiment, and summarize the ASG field test results.

II. NASA Haughton-Mars Project The Haughton-Mars Project (HMP) is an international multidisciplinary field research project dedicated to advancing planetary science and exploration. The HMP was established in 1997 and is centered on the scientific study of the Haughton meteorite impact crater and surrounding terrain on Devon Island, High Arctic, viewed as a planetary analog, in particular for the Moon and Mars.1 Devon Island is the largest uninhabited island on Earth. The environment on Devon is extreme by terrestrial standards and is best described as a polar desert (not tundra), i.e., cold (-40°C < T < +10°C), dry, and unvegetated. The island presents the single largest continuous area of barren rocky polar desert on Earth. Devon Island is home to Haughton Crater, a 20 km-diameter meteorite impact crater formed 23 Ma ago, during the Miocene epoch. The HMP site has been used extensively by NASA as a uniquely relevant Moon and Mars science and exploration operations analog. The site is commonly referred to as “Mars On Earth”. Research at HMP is divided into two programs: Science and Exploration. The HMP Science program seeks to learn about the site’s geology and biology, in order to gain insights into the nature and evolution of the Moon, Mars, and other planetary bodies via comparative studies.2 In the process, the HMP Science program also contributes new knowledge about Devon Island, the Arctic, and the evolution of the Earth through time. The HMP Exploration program seeks to use the site to develop, test, and validate new exploration technologies and strategies for planning the future human and robotic exploration of the Moon and Mars. Exploration systems studied include habitats, spacesuits, ground vehicles, aircraft - drones and other unmanned aerial vehicles (UAVs) -, robotic rovers, drills, instruments, tools, life support systems, plant growth systems, and communications and other information systems. Human factors and crew resource management (CRM) studies are also carried out. Research at HMP is supported by NASA and other research partners in government, academia, non-profits, and industry. 2 International Conference on Environmental Systems

The Haughton-Mars Project Research Station (HMPRS), the HMP’s Base Camp on Devon Island, is located at 75° 26' N, 89° 52' W, in the northwestern rim area of Haughton Crater (Fig. 1). The HMPRS is currently the largest privately operated polar research station on Earth, and the only one dedicated to planetary analog science and exploration studies.3 The NASA HMP is headquartered at NASA Ames Research Center (ARC) at Moffett Field, California. For more information: www.marsonearth.org

Figure 1: Haughton-Mars Project Research Station: Left: Location map. Center & Right: Base camp. (HMP).

III. EVA on the Moon and Mars: The Human-Machine Interface Challenge EVA spacesuits are both enabling and limiting. Although dexterous manipulation and one-handed or two-handed manipulation at a task site are considered advantages of EVA, dexterity remains significantly restricted.* Because pressurization results in stiffening of the pressure garment, an astronaut’s motions and mobility are significantly restricted during EVAs. Spacesuits also limit as astronaut’s situational awareness because of the directionally- restricted and optically-degraded visibility imposed by the helmet and visor.

A. Spacesuit Gloves The limited dexterity allowed by pressurized spacesuit gloves is a significant and well-recognized problem. The NASA-STD-3000 standards document states: “Space suit gloves degrade tactile proficiency compared to bare hand operations…Attention should be given to the design of manual interfaces to preclude or minimize hand fatigue or physical discomfort.”4 Astronauts are commonly on record identifying spacesuit gloves as a top priority item in their EVA apparel needing significant improvement (Fig. 2). Apollo astronaut-geologist Harrison “Jack” Schmitt, who wore the A7-LB suit on Apollo 17, singled out hand fatigue and dexterity as the top two problems to address in EVA spacesuit design for future Moon and Mars exploration.5 While the difficulty of using pressurized suits and gloves may be well mitigated in EVA operations outside spacecraft in orbit (e.g., on the ISS), as optimally-adapted tools can be developed for narrowly-constrained tasks, EVA operations on planetary surfaces, including on the Moon, Mars, and small bodies such as the of Mars, and , require interacting with a wide range of unknown terrain, materials, and conditions, which make having optimally adapted tools for all circumstances impossible to achieve in practice. Beyond the challenges of awkwardness, discomfort, and constant effort in interacting with extra-vehicular environments in pressurized spacesuits, astronauts are also at significant risk of injury. Hand and upper-extremity overuse and repetitive injuries in astronauts have been and continue to be a common problem in EVAs.6 The constraints and pressures of spacesuits and gloves have been shown to negatively impact upper-extremity function in ways that can rapidly result in overuse/repetitive injuries, which will then negatively impact mission productivity. Beyond an astronaut’s fingers, each one of his/her hands, wrists, arms and shoulders may also experience significant fatigue when used in repetitive motions and/or when held in fixed positions for extended periods of time against a pressurized spacesuit’s neutral posture.

* The NASA-STD-3000 Man-Systems Integration Standards Vol. 1 document contains human-systems integration design considerations, design requirements, and example design solutions for development of manned space systems. Section 14 focuses on EVA. Section 14.1.2.1.2 covers Advantages of EVA. Section 14.1.2.1.3 Limitations of EVA (Ref. 4).

3 International Conference on Environmental Systems

Figure 2. EVA Pressurized Spacesuit Restrictions. Left: Astronaut Charlie Duke (Apollo 16) having difficulty bending down to pick up a tool. Center: Astronaut-geologist Jack Schmitt (Apollo 17) has a precarious single-hand grip on just half of the lunar rake’s handle bar due to the stiffness of his suit and glove. His arm positions also suggest difficulty bringing both hands close together at waist level. Right pair: Schmitt’s right hand is used to lever the lunar rake instead of gripping it firmly, illustrating difficulty with gripping in a pressurized glove (NASA, Apollo 16 & 17).

Meanwhile, future plans for space exploration include endeavors that will continue and even increase the demands on the hand and upper extremity.6 In addition to an expected ramping up of EVA time once lunar or surface operations begin, there is the expectation that astronauts will need to operate a wide variety of science and exploration instruments, tools, machines, and other surface systems, including human-operable robotic assistants during EVAs. Examples of the latter may include robotic arms, drills, trenchers, dirt-movers, mobile communications assets, transportation vehicles and other mobile platforms. While many of these, in their current terrestrial commercial off- the-shelf (COTS) versions, may come with relatively intuitive and simple user interfaces, some are relatively complex to use and would be a major challenge to operate while wearing a pressurized spacesuit. In this context, unmanned aerial vehicles (UAVs) or drones present an epitomic challenge, as they are generally operated via relatively complex human-machine interfaces requiring sustained hand-eye coordination, - therefore good situational awareness -, and the use of both hands, including of several fingers on each – therefore good dexterity. How might an astronaut in EVA operate a remote-controlled flyer in real-time if his/her pressurized spacesuit were to remain as restrictive as it is now? The NASA-STD-3000 document (Sec. 14.3.2.1 on Spacesuit Design Considerations & Dimensions) lists as a key design goal for EVA suits viewed as complete anthropometric systems, goal “d”: “Glove dexterity – Space suit gloved hand dexterity that approaches that of bare-hand operations.”4 To meet this challenge, spacesuit gloves have to either be significantly improved in dexterity (Approach A), and/or be rendered usable more or less as they are, but in ways that minimize workload (Approach B). We aim to investigate if/how the ASG might offer an Approach B solution.

B. Augmented Reality and Voice-Controlled Human-Machine Interfaces One way to enhance situational awareness and reduce requirements on physical action by astronauts is to consider the use of human-machine interfaces (HMI) that include head-up displays (HUDs) and speech-recognition-based controllers. The importance of creating such EVA enhancing in-suit capabilities was also recently emphasized by Apollo 17 astronaut Jack Schmitt.5 Early field operational trials of in-suit wearable computers with head-mounted displays (HMDs) as initial implementations of HUDs, and voice control were carried out at HMP in collaborations between Collins Aerospace (Hamilton Sundstrand then), Mars Institute, SETI Institute, NASA ARC, and Simon Fraser University.7-9 Wearable computers, voice controllers, and fixed cameras were successfully integrated to concept spacesuits and used in simulated EVAs to access science operations data (map layers) in real-time, and log field observations (Fig. 3). These initial EVA HMI studies held great promise, but still faced two important technology maturation challenges: a) the development of an in-helmet HUD interface that presents no risk of physical injury to the astronaut or any risk of inadvertent mechanical misalignment with the astronaut’s field of view or line of sight; b) the development of voice-recognition software capable of reliable speech recognition within (in spite of) the inherently noisy life support system (LSS) environment of a spacesuit. 4 International Conference on Environmental Systems

Figure 3. Early EVA HMI Field Tests at NASA HMP in the Arctic (2000-2004). Left: Xybernaut wearable computer. Center left: Training speech-recognition software prior to simulated EVA. Center right: Wearable computer, voice controller and camera integrated to concept spacesuit. Hardware exceeds volumetric constraint of helmet. Right: Over time, hardware was reduced in size to fit inside helmet bubble volume. (HMP/NASA).

IV. Drone / UAV Operations on the Moon and Mars Unmanned aerial vehicles (UAVs), or drones, have experienced a historic surge in commercial availability and popularity in a wide range of terrestrial applications. They have become an important tool in field research, logistics, and safety, and also in support of outreach activities (esp. aerial photography and filming). In field geology and biology, drones are having a game-changing positive impact. They allow capturing reliably, quickly, and affordably, for both real-time and post-flight analysis, aerial imaging and other remote-sensing data on a field site, e.g., context imaging for geology. Drones are also used to explore and study extreme or otherwise hard-to-access environments, including to collect samples, allowing field scientists to gain important and often unique insights about their field site. Drones are anticipated to be an integral part of future field exploration tools for human exploration of the Moon and Mars. Although the lunar atmosphere is too tenuous to fly any aircraft, and so is the Martian atmosphere over high altitude terrain, gas thrustered drones have been proposed as a potential solution in those settings.10-12 Ideas of operating aircraft on Mars are not new. Although Mars’ surface atmospheric density at lower elevations is of order 80 times less than on Earth - 0.015 kg.m-3 vs 1.2 kg.m-3 -, aircraft may de designed to fly on Mars, with 2 Mars’ gravity being only 0.38 gEarth. As lift is proporational to rv A where r is atmospheric density, v is airspeed, and A is lifting area, sufficient lift may be efficiently achieved by increasing mainly airspeed. In addition to Mars airplanes,13 Mars rotorcraft, unmanned and manned, have been proposed and also show promise.14-16 The NASA Jet Propoulsion Laboratory’s (JPL) Mars helicopter on NASA’s Perseverence () mission has counter-rotating coaxial rotors spinning at up to 2800 rpm, a factor of 4-9 times higher than the main rotors on most terrestrial drones and crewed helicopters, with somewhat large form factors (1.2 m tip to tip; max chord length ~0.13 m) to keep blade tip speeds below Mach 0.7.16-18 NASA’s Dragonfly mission entails flying a UAV on Saturn’s moon . -3 19,20 Titan, where atmospheric density is 1.88 kg m , gravity is 0.14 gEarth, and temperature is 94.2 K (-179°C, -290°F). Field studies of the applications, design, and operations of UAVs for Mars have been carried at the NASA HMP since the earliest days of the project. Highlights include field tests of Carnegie Mellon University’s robotic helicopter in 1994,21 field studies of aerial imaging survey requirements using various NASA ARC planetary analog aerial explorers in 2002 and 2003,22 and field tests of NASA Langley Research Center’s (LaRC) Mars Electric Reusable Flyer (MERF).13,23 While many autonomous aircraft have been considered for Mars exploration, and the Ingenuity Mars helicopter represents a first technology demonstration of such an aircraft to test powered flight on Mars, astronaut-deployed drones are anticipated to also be an important application of UAVs in future Mars exploration.23 Astronaut-deployed drones could be used in support of a wide range of surface science and exploration activities, including surveying, mapping, examining, sampling, scouting, searching, rescuing, fetching, filming, and inspecting.23 While these aerial operations can be carried out with the astronaut(s) in intra-vehicular activity (IVA) mode, i.e., in shirt-sleeves and located inside a pressurized habitat or rover, it will be essential to also be able to carry them out while astronauts are on EVA. Sampling of a site of astrobiological interest, for instance, may require an astronaut’s direct line of sight to both the drone and the sampling target, and the ability to select a specific sample with deliberate real-time astronaut control inputs. Scouting, fetching, and search and rescue during EVAs are also operations that may require real-time astronaut control inputs.

5 International Conference on Environmental Systems

V. Astronaut Smart Glove: Origin, Design, and Initial Development

A. Origin of the Astronaut Smart Glove The start-up technology company Ntention, formerly Arveng Technologies, invented and developed the “Smart Glove” HMI upon which the ASG is based, for intuitive, single-handed operation of commercial drones and other complex agent systems. The idea of a Smart Glove for HMI was conceived in February 2016, as an evolution from the original 2015 idea of a Dance Glove, capturing motion through sensors embedded in a glove. The first prototype iteration was a Drone Glove, created to control commercial drones, a proof-of-concept showcase to demonstrate the abilities of such a control mechanism for use as an intuitive tool and an eventual replacement for the cumbersome joysticks and controller solutions currently in widespread use in the commercial drone industry. The concept of a Smart Glove for HMI matured significantly in 2017 as the idea progressed through several prototype stages, and the creators realized that the potential of this HMI technology was far greater than just commercial drone control. By 2018, after 14 prototyping iterations and a broadening of applications, Ntention had created gloves to control music and robotic arms, and identified potential applications in Augmented Reality/Virtual Reality (AR/VR), in addition to the original Drone Glove. During the Fall of 2018, a “BETA* Edition” of the Drone Glove was successfully crowdfunded by early adopters in the consumer electronics market, but further analysis showed that this was a capital intensive and high-risk , with predictions of only medium to low returns. The development of the Smart Glove concept continued, but the company changed its strategy to focus on delivering business-to-business (B2B) solutions. For more information: www.ntention.com HMP PI P. Lee witnessed a demonstration of Ntention’s technology during the Energy Valley 2019 conference in April 2019 in Oslo, Norway, and suggested that the Ntention Smart Glove might offer a viable HMI solution to enabling astronauts on EVA on the Moon, Mars, and elsewhere in space, to operate robotic systems, including drones, in spite of the encumberance and rigidity of pressurized spacesuits. A collaboration to further investigate this “Astronaut Smart Glove” (ASG) concept was established between researchers at Mars Institute, SETI Institute, NASA Ames Research Center, Collins Aerospace, and Ntention, with Ntention COO M. Tamuly taking on the challenge of leading the redesign and integration of the Ntention Smart Glove into a first ASG prototype. The concept would be an expansion on the existing Ntention Smart Glove technology for drone operation, in particular with the addition of vision-based control to the original strictly gesture-based control. A HMD in the form of AR-glasses was integrated to the system as a means of simulating the fully integrated HUD that could be implemented in later generations of spacesuits. Development moved forward rapidly, and plans were made to field test and demonstrate the concept during HMP-2019 field campaign on Devon Island, High Arctic, in August 2019. The ASG field test would use a Collins Aerospace Moon/Mars analog concept spacesuit already undergoing field evaluations at the NASA HMP.

B. Astronaut Smart Glove System Description The first generation ASG System hardware tested at HMP-2019 consists of three main subsystems: a sensor glove, a set of AR-glasses, and a processing device. The subsystems work together to create a complete system that removes the need for conventional controllers, which is critical for any EVA spacesuit application (Figs. 4 and 5). The sensor glove uses a carefully selected set of sensors to accurately capture the movements of the user’s hand, before they are wirelessly transferred to the processing device. The sensor glove is designed around an EVA glove liner that fits inside the Collins Aerospace concept spacesuit, retrofitted with the necessary sensors and electronics. The AR glasses used in the field tests are a pair of Epson Moverio BT-30C. These glasses feature high definition OLED-displays, which make it possible to fly a drone purely based on the real-time video feed from them. An important feature of the glasses is that they have an embedded Inertial Measurement Unit (IMU). This means that data describing the orientation of the user’s head can also be transferred to the Processing device. The processing device is the computer at the core of the system, responsible for data processing and communication with the other subsystems. It is connected wirelessly to both the glove and the drone, while the AR-glasses use a wired connection. The processing device receives motion data from the sensor glove and translates these into commands for either the visual system or the drone depending on the context. In addition to the glove data, the system utilizes the the IMU data (orientation of the user’s head) from the AR-glasses to further improve the interaction. The visual interaction system displayed on the AR-glasses allows the user to navigate menus by using simple hand gestures. By “grabbing and pulling” virtually with a hand, the user is able to scroll throughand select control options such glove calibration and drone control. Such an interaction system is highly intuitive and simple, and consistent with operating within the optimal work envelope of a spacesuited astronaut in EVA.

6 International Conference on Environmental Systems

Figure 4: The Astronaut Smart Glove (ASG). Left: EVA glove liner with integrated Ntention smart glove system, including stretch sensors and “GloveBox” containing power supply, additional hand motion sensors, and wireless transmitter. Center: ASG system integrated to EVA glove, with control buttons on back side of hand accessible via other hand. Right: In addition to the glove itself, the ASG system includes an augmented reality (AR) head-mounted display, head motion sensors allowing the drone camera gimbal to be controlled by head motions, and the drone controller interface. L: M. Tamuly, Ntention COO; R: S. Tagestad, Ntention Engineer. (HMP/CollinsAero/Ntention).

Figure 5: Left: AR FPV View in IVA Mode. Right: ASG Functional Block Diagram. The Ntention Smart Glove motion data output can be modulated by adjusting the glove’s sensitivity to hand and finger motions (Ntention/HMP).

The ASG system’s information display interface is highly streamlined. The total amount of data within the ASG system is too excessive to be shown all at once in a visual feed. A cluttered visual information display would increase cognitive load for the operator, thereby increasing response time, impacting performance, and possibly even compromising safety (that of the astronaut, other crew, the drone, and/or other assets). It is critical that the operator receive only relevant data at the right time, facilitating appropriate and timely decision-making. When operating in an AR environment, there is also the possibility of experiencing unwanted occlusion in the visual field. In order to reduce the potential complications of cluttered visualizations, menus were constructed minimalistically, and only relevant information is displayed during flight operations. ASG system menus were deisgned with intuitive ease of use in mind, as the interaction model is designed to closely match familiar interactions with interfaces such as physical buttons.

VI. Astronaut Smart Glove: HMP-2019 Field Test The HMP-2019 ASG field experiment, with the ASG integrated to an unpressurized concept spacesuit, comprised two main tests: Test A: an operational field test of the ASG to evaluate its ability and performance in allowing an astronaut to execute a variety of basic field science and exploration tasks while keeping hand gestures within ranges of motion limits allowed had the suit been pressurized; and Test B: a standardized flight test designed to evaluate quantitatively the handling qualities of a drone using the ASG. Field team personnel, roles, and responsibilities are listed in Table 1.

Table 1. HMP-2019 Astronaut Smart Glove Field Test Field Team Field Participant Primary Role Responsibility ASG Principal Investigator (PI) Coordination of ASG Field Test, Data & Video Capture, Data Analysis. Moina Tamuly ASG Development Lead Lead for Test-A. Secondary Suit Subject. Drone to Drone Video Capture. Sondre Tagestad ASG Engineering Lead Lead for ASG Hardware/Software. Secondary Suit Subject. Drone Safety Officer. Brandon Dotson ASG Test Pilot Lead for Test-B. Primary Suit Subject. John W. Schutt Exploration Field Ops Consultant HMP Base Support In addition to the above HMP-2019 ASG Field Team, Chris McKay at NASA ARC, Greg Quinn, Tom Chase and Jake Rohrig at Collins Aerospace, and Magnus Arveng, Haakon Pettersen, and Frank Oygard at Ntention were key participants in the HMP-2019 ASG Experiment. 7 International Conference on Environmental Systems

A. HMP-2019 ASG Test A: Operational Field Test of the Astronaut Smart Glove For Test A, the following Moon/Mars drone-based science and exploration tasks were investigated: surveying, examining, sampling, scouting, fetching, and inspecting. Each task was performed from drone takeoff from, to drone landing at, a base point located < 3 m of the suited drone operator. Maximum drone range from the base was < 0.5 km to limit flight time for each task iteration to < 5 min. Surveying entailed flying the drone at 50 to 400 ft AGL (above ground level) and imaging the surrounding terrain in search of targets of interest (TOIs). Examining entailed flying to, and observing at close range (< 3 m), a specific TOI, e.g., a fossiliferous outcrop, a boulder, or a fracture/joint. Sampling entailed having the drone touch down at, and lift off from, a TOI from which representative material was sought. Scouting entailed flying the drone at 50 to 400 ft AGL and searching for, and where possible validating, trafficable access routes to reach distant TOIs. Fetching entailed dispatching a drone to a parked ATV (all-terrain vehicle) to simulate retrieving a needed item (e.g., life support supplies, rescue or survival gear) from a distant field asset (e.g., vehicle, habitat, cache), and bringing it back to the astronaut. Inspecting entailed assessing the condition and operability of a field asset (e.g., vehicle, habitat, or cache) with the drone operator in IVA mode. All tasks considered were carried out in partially simulated EVA mode, except inspecting, which was done in IVA mode. The nature and amplitude of the hand gestures needed to fly the drone in the performance of each task were recorded. These gestures included hand gripping and extension, wrist adduction and abduction (wrist “pitch”), wrist flexion and extension (wrist “yaw”), and wrist rotation (wrist “roll”), with amplitudes of motion measured from an arbitrary neutral starting position chosen to be the “flying” hand’s forearm extending forward and positioned horizontally within the 5th percentile optimum two-handed work envelope of a Shuttle/ISS EMU-class (Extra- vehicular Mobility Unit) suit.4 Although the latter is a micro-gravity suit and not a planetary surface suit designed for gravity field operations, NASA’s EVA Office has suggested using the work envelope requirements of the current Shuttle/ISS EMU as a starting assumption for the work envelope of future xEMU-class suit systems. Using for now the limited 5th percentile optimum work envelope required for the current EMU suit represents a conservative assumption for the likely larger work envelope of future higher mobility planetary suits, and one adequate for our test. Drone operation control inputs via the ASG are designed to be intuitive and simple, including in flight. From the initial neutral hand position and attitude defined for the drone to be in a stable hover and facing forward, hand gestures in different directions result in the drone flying in the corresponding directions. Thumb inputs are set to control yaw. Tightening the fist (gripping) with the hand in its neutral position initiates vertical landing. How specific gestures result in specific drone control inputs is changeable to suit user preferences, including via real-time reprogramming. What defines the initial neutral position is also changeable, a feature that may eventually help alleviate arm and shoulder fatigue from holding any single position in an pressurized suit for an extended time. The ASG was field tested outside the HMPRS, but within 1 km of base camp to optimize field test safety. The test location, at 75°25’53”N, 89°52’27”W, was at the southwestern end of Husband Hill, on the south side of Pete Conrad Valley, a site offering a wide range of topographic slopes (0-180°, i.e., from horizontal to inverted overhangs), geologic features of interest and of potential relevance to Mars (e.g., Paleozoic fossils of algal colonies), a micro-oasis for biological sampling, and a maze-like network of small valleys requiring scouting (Figs. 6 and 7).

Figure 6. Astronaut Smart Glove Field-Test at the HMP Moon/Mars Analog Site on Devon Island, High Arctic. Left: Subject in Collins Aerospace concept suit flying drone with ASG. Right: Suit camera view. (NASA / HMP).

Figure 7. Moon/Mars Science & Exploration Ops Tasks Evaluated Using ASG-Operated Drone at HMP-2019. 8 International Conference on Environmental Systems

Each task was successfully executed 2 to 5 times over the course of two days for Test A. B. Dotson, a US Army rotorcraft Test Pilot, served as primary suited test subject. M. Tamuly served as secondary suited test subject. S. Tagestad served as additional suited test subject during pre- and post- EVA ASG system calibration work. For each task iteration, the nature and amplitude of ASG hand gestures was recorded via real-time notes and continuous video capture. Whenever initial hand or wrist motion ranges would be estimated to exceed the range of motion allowed by a pressurized (EMU) spacesuit’s work envelope, the ASG sensitivity to hand/wrist motions was increased until the drone could be flown with all gesture amplitudes fitting well within the EMU’s work envelope (Figs. 8 and 9). Adjustments of sensitivity settings were done by the “free” hand reaching over to the control buttons on the ASG “GloveBox” on the flying hand within the 5th percentile work envelope.

Figure 8: Left: ASG ranges of motion for drone operation are small, consistent with plausible motion ranges for a pressurized suit. The largest gesture amplitude recorded, for wrist abduction/adduction, was <20°. Right: ASG hand gesture sensitivity is adjustable to minimize motion range needed for drone control (HMP / Ntention).

Figure 9. Astronaut Smart Glove Work Envelope: Left: NASA’s specification of EVA crewmember optimum work envelopes (NASA-STD-3000) is used a conservative starting assumption for the work envelopes of future xEMU class suits (EVA-EXP-0035). Right: The ASG work envelope fits within the 5th percentile optimum two-handed work envelope, allowing for optimum ease of motion for each hand working individually or both hands working together. The ASG GloveBox control buttons on the back of the “flying” hand are readily reached by the “free” hand.

9 International Conference on Environmental Systems

Post-test analysis of the video footage data set shows that the largest gesture amplitudes recorded at intermediate hand motion sensitivity setting were ≤ 20° for wrist abduction/adduction.24 Other gesture amplitudes were ≤ 15°. Post- EVA debriefs of the suited test subjects included voice-recorded interviews to capture qualitative feedback on their experiences using the ASG and the completion by each of a questionnaire using a Cooper-Harper rating scale25 to quantify these experiences (Fig. 10 Left). Ease of use of the ASG in a) tracking the drone in flight (via direct line of sight or the AR HMD interface) and b) in performing Test A tasks, were rated at 2 (acceptable effort and comfort) and 3 (moderate difficulty or discomfort), respectively. Further quantitative assessment of the adequacy of the AR HMD was carried after both Tests A and B (see below). Test A confirmed the intuitive ease of use of the ASG HMI system, the acceptable situational awareness it affords, and its ability to keep hand gestures within the EMU’s work envelope in this intial field test using an unpressurized suit to perform a range of Moon/Mars science and exploration tasks.

B. HMP-2019 ASG Test B: Astronaut Smart Glove Standardized Drone Handling Test For Test B, the main objective was to evaluate quantitatively the handling qualities of a drone using the ASG by comparing handling performances obtained by the drone operator in the following specific, distinct conditions: 1) Operator in unpressurized spacesuit using the ASG with AR display; 2) Operator in unpressurized spacesuit using a conventional dual joystick handheld drone controller; 3) Operator in “shirt-sleeves” using the ASG with AR display; 4) Operator in “shirt-sleeves” using a conventional (dual-joystick) drone control interface. These four unpressurized configurations serve to establish a quantitative baseline with which later handling qualities tests using a pressurized spacesuit may be compared. We used classical flight test techniques and standardized handling qualities tests used by tests pilots of crewed rotorcraft to assess drone handling performances, including the precision of rotorcraft UAV control. Adapted mission task elements (MTEs) from the Aeronautical Design Standard for Handling Qualities Requirements for Military Rotorcraft (ADS-33E-PRF) were applied. A specific flight course (path) was established at the HMPRS base camp, with specific MTEs to be performed during each flight “run” including pickup and landing, flight legs with left and right crosswinds, and left and right yaw turns, including a midpoint hovering turn (Fig. 10). The course was flown as precisely as possible multiple (2 to 3) times under each one of the above conditions. For each run, each MTE was timed and its accuracy of performance noted in real-time and also captured continuously on video. Post-flight analysis of vehicle flight path and landing accuracy using Tracker Video Analysis and Modeling Software allowed handling performance to be objectively and quantitatively assessed. The performance of each task was then assigned a score, resulting in the overall flight run being also assigned a score. Ease of flight handling and workload was also evaluated quantitatively in each configuration using a Cooper-Harper rating scale25 and a Bedford workload rating scale. Test B details may be found in a companion paper by Dotson et al.26

Figure 10. Left: Cooper-Harper rating scale. Center: Flight test course at HMPRS. Right: Flight path and landing accuracy measurements using tracker video analysis & modeling software (HMP/B. Dotson).

Adequate flight handling and landing accuracy performances were achieved for all of the examined configurations. However, configurations 2 and 4 with the conventional dual-joystick controller interface were significantly more demanding workload-wise, both physically and mentally, suggesting that use of such a controller interface will likely lead to significant deterioration of handling performance when our testing transitions to a pressurized suit in the future. The much greater workload comfort associated with using the ASG in configurations 1 and 3 suggest greater margin for performance deterioration when flight operations include a pressurized suit, opening the prospect that the ASG concept might offer a pathway to technologic maturation for EVA systems.

10 International Conference on Environmental Systems

C. HMP-2019 ASG Augmented Reality (AR) Head-Mounted Dispay (HMD) Evaluation As part of both Tests A and B, interface controls and displays were specifically evaluated using an adaptation of Aircrew Station Controls and Displays for Rotary Wing Aircraft (MIL-STD-250) and Department of Defense Design Criteria Standard: Human Engineering (MIL-STD-1472G). Deficiency ratings were assigned in accordance with the NAVAIR Test Reporting Handbook.26 The ASG and associated AR HMD, as integrated to an unpressurized concept spacesuit and tested in the field, rated favorably. Because the AR HMD within the glasses is transparent, the operator is able to monitor the video feed from the drone while still remaining in direct line-of-sight visual contact with the flyer, and also adequately aware of his/her surroundings. The ASG and AR HMD interface also allowed the drone to be flown under full control of the operator even in strict FPV mode, when the drone was out of direct line of sight.

Conclusion

Drones are anticipated to be among the wide array of robotic systems that will be available to astronauts in science and exploration operations on the Moon, Mars, and beyond. Although their capabilities will be multidimensional and complex, their operation may be rendered possible even by astronaut operators confined to pressurized spacesuits thanks to advanced HMIs. The Astronaut Smart Glove (ASG) system, including its AR HMD and software system, seamlessly map hand motion closely to the motion of the drone in flight, such as roll, pitch, and yaw, and head motion to the drone camera gimbal. This HMI allows the drone operator to operate a drone intuitively, efficiently, productively, and safely, with minimal training. Our field tests at HMP-2019 yield promising, if preliminary, results. Test A suggests that several key drone-based science and exploration tasks may be performed with adequate situational awareness and ASG gesture amplitudes allowed by conservative EVA spacesuit work envelopes. While conducting field tests with an unpressurized suit to learn how an astronaut might use a pressurized spacesuit can be fraught with shortcomings, in particular because a pressurized suit is far more rigid and will impose severe constraints on actual range of motion, dexterity, and fatigue associated with performing gestures repeatedly and/or holding specific positions off a pressurized suits’s neutral posture, Test A’s scope was limited to showing how the ASG allows operating single-handedly a robotic asset as complex as a drone to perform a wide range of Moon/Mars relevant science and exploration tasks in spite of the restricted situational awareness afforded by an EVA suit, and with gesture amplitudes fitting well within the work envelope of the current EMU suit. Translation from hand motions to drone commands was done in a manner that enables both intuitive and precise control. The sensitivity of the different commands were adjustable to allow the suit subject to fly the drone while remaining well within pressurized suit hand motion and dexterity limits. Test B suggests that the ASG might also allow precision handling and flying of drones from a pressurized suit without imposing unreasonable workloads, whereas the use of a conventional drone interface would likely be a show- stopper. Tests A and B also show that the ASG and associated AR HMD would allow the drone to be operated without the suit subject actually seeing the drone directly. The video feed from the drone camera, which can optionally be seen in the AR-glasses, was sufficient for the full range of drone operations investigated. The initial field study findings reported here do not qualify (and were not expected to) the ASG system for EVA use, but they may allow proposing and planning next steps of work that will require significantly more resources, including testing with pressurized suits. Future possibilities may also lie in giving even more control opportunities and flexibility to the operator by enabling more intuitive interaction systems through the glove in an EVA suit.27 The ASG offers potentially a novel way of achieving goal “d” of NASA STD-3000 Sec. 14.3.2.1 on glove dexterity, which is to enable “space suit gloved hand dexterity that approaches that of bare-hand operations”, by offering a solution via Approach B described in Section III.A.4,28,29 The Smart Glove technology renders “bare-hand” operations of even a complex robotic system such as a drone so simple that even a spacesuit glove’s limiting hand motion and dexterity might allow “bare-hand” operations to be matched. The human sensory system is highly adaptive. A well designed HMI system can enable astronauts to efficiently and productively process large amounts of data and execute multiple complex science and exploration tasks, such as flying a drone while analyzing the imaging data it returns in real-time. Ultimately, the ASG helps open the prospect of creating highly intuitive, seamless, capable, and safe HMIs that will help minimize workload and injury to astronauts and maximize productivity and safety as they interact with a wide range of robotic systems for science and exploration on the Moon, Mars, and beyond.

11 International Conference on Environmental Systems

Acknowledgments The ASG field study was carried under the auspices of the NASA HMP as part of Cooperative Agreement NNX14AT27A between NASA and the SETI Institute (PI P. Lee). Support was provided by Mars Institute, SETI Institute, NASA ARC, Collins Aerospace, and Ntention. B. Dotson participated in this study as the recipient of the 2019 HMP Apollo Fellowship Award. The HMP and Ntention are grateful to their respective sponsors and supporters.

References

1Lee, P. (2010). Haughton Crater and Devon Island. In Planetary Analogs, P. Doran et al. Eds. Springer-Verlag, 278-283. 2Lee, P. and G. R. Osinski (2005). The Haughton-Mars Project: Overview of science investigations at the Haughton impact structure and surrounding terrains, and relevance to planetary studies. Meteor. Planet. Sci. 40 (HMP Special Issue), 1755-1758. 3Lee, P., et al. (2016). Haughton-Mars Project Research Station (HMPRS), Devon Island, High Arctic: A planetary science and exploration field research facility. Lunar Planet. Sci. Conf. 2016, 21-26 March 2016, The Woodlands, TX. #3073. 4NASA STD-3000 (1995). Man-Systems Integration Standards. Extravehicular Activity (EVA). EVA Mobility and Translation. Vol. 1, Sec. 14. https://msis.jsc.nasa.gov/sections/section14.htm 5Schmitt, H. H. (2018). Human efficiency improvements in lunar and Martian field exploration. NASA SSERVI Exploration Science Forum 2018, June 25, 2018. 6Viegas, S. F., D. Williams, J. A. Jones, and S. Strauss (2004). Physical demans and injuries to the upper extremity associated with the space program. J. Hand Surgery 29, 359-366. 7Boucher, M. F., E. Hodgson, S. K. Murray, P. Lee, and S. Braham 2002. Investigation of EVA information technology in a Mars analog arctic field science setting. ICES Conf., 02ICES-213. 8Hodgson, E et al. (2003). Requirements and potential for Enhanced EVA Information Interfaces. SAE 2003-01 2413, 7/2003. 9Hodgson, E et al. (2011). Evolutionary Development of Exploration EVA Systems Design and Operations Through Analog Field Tests: Lessons from the NASA Haughton-Mars Project, 2008-2010. AIAA ICES-2011, Portland, OR. 17pp. 10Lee, P. (2019). Habitability of lava tubes on the Moon and Mars: Lessons from Earth. Final Summary to 2018 MBR Space Settlement Challenge, Dubai Future Foundation, Guaana.com. 11Lee, P. (2020). Ice-rich caves on the Moon and Mars: Prospects and pragmatic recommendations for exploration. 3rd Int’l. Planet. Caves Conf., 18-21 Feb 2020, San Antonio, TX. #1066. 12Lee, P., E. Kommedal, A. Horchler, E. Amoroso, K. Snyder, & A. F. Birgisson (2019). Lofthellir lava tube ice cave, Iceland: Subsurface micro-glaciers, rockfalls, drone LiDAR 3D-mapping, and implications for the exploration of potential ice-rich lava tubes on the Moon and Mars. LPSC-2019, The Woodlands, TX. #3118. 13North, D., S. Geuther, P. Lee (2020). Mars Electric Reusable Flyer: A Mars near-surface exploration flight vehicle. In prep. 14Young, L. et al. (2002). Rotorcraft as Mars Scouts. Proc. IEEE Aerospace Conf. 2002, Vol. 1, 1 - 378. 15Young, L. A., Lee, P., Briggs, G., and Aiken, E. (2005). Mars Rotorcraft: Possibilities, Limitations, and Implications for Human/Robotic Exploration. Proc. IEEE Aerospace Conf. 2005, Big Sky, MT, 2005, 300-318. 16Balaram, J. and P. T. Tokumaru (2014). Rotorcrafts for Mars Exploration. 11th International PlanetaryProbe Workshop. 17Balaram, B., et al. (2018). Mars Helicopter Technology Demonstrator. AIAA Atmospheric Flight Mechanics Conf. 2018. 18Koning, W. J. F., W. Johnson, and B. G. Allan (2018). Generation of Mars helicopter model for comprehensive analyses. AHS Specialists’ Conf. on Aeromechanics Design for Transformative Vertical Flight, San Francisco, CA, USA, Jan. 16-18, 2018. 19Lorenz, R. D. et al. (2018). Dragonfly: A Rotorcraft Lander Concept for Scientific Exploration at Titan. Johns Hopkins APL Technical Digest, Vol. 34, No. 3, 374-387. 20Turtle, E. et al. (2020). Dragonfly: In situ exploration of Titan’s organic chemistry and habitability. 51st LPSC, Abs. #2288. 21Miller, R., O. Amidi, and M. Delouis (1999). Arctic test flights of the CMU autonomous helicopter. Proc. Assoc. Unmanned Vehicle Systems International 1999, 26th Ann. Symp. Baltimore, M.D. 22Pisanich, G., L. Plice, C. Ippolito, L. A. Young, B. Lau, and P. Lee (2004). Initial efforts toward mission-representative imaging surveys from aerial explorers. Proc. Real-Time Imaging VIII, Vol. 5297. 23Lee, P., D. North, S. Geuther (2020). Haughton-Mars Project: Investigation of Mars UAV science and exploration operations using the NASA Mars Electric Reusable Flyer. In prep. 24Dianat, I., C. M. Haslegrave, and A. W. Stedmon (2012). Methodology for evaluating gloves in relation to the effects on hand performance capabilities: a literature review, Ergonomics 55, 1429-1451. 25Cooper, G. & R. Harper (1969). The Use of Pilot Rating in the Evaluation of Aircraft Handling Qualities. NASA-TN-D5153. 26Dotson, B. N., P. Lee, J. Schutt, S. Tagestad, and M. Tamuly (2020). Handling qualities flight test techniques for astronaut- operated, unmanned aerial vehicles for Mars exploration. AIAA 2020-3092. 27Newman, D. (2000). Life in extreme environments: How will humans perform on Mars? Gravit. Space Biol Bull 13, 35-47. 28Main, J. A., S. W. Peterson, and A. M. Strauss (1994). Design and structural analysis of highly mobile space suits and gloves. J. Spacecraft and Rockets 31, 1115-1122. 29Mousavi, M. M. S., E. P. Ambrosio, S. Appendino, F. C. Chen, A. Favetto, D. Manfredi, F. Pescarmona, and A. Soma (2011). Spacesuits and EVA gloves evolution and future trends of extravehicular activity gloves. 41st ICES, 17-21 July 2011, Portland, OR, AIAA 2011-5147. 12 International Conference on Environmental Systems