Reducing Cognitive Load and Improving Warfighter Problem

fpsyg-11-554706 November 11, 2020 Time: 15:25 # 1

ORIGINAL RESEARCH published: 17 November 2020 doi: 10.3389/fpsyg.2020.554706

Reducing Cognitive Load and Improving Warﬁghter Problem Solving With Intelligent Virtual Assistants

Celso M. de Melo1*, Kangsoo Kim2,3, Nahal Norouzi4, Gerd Bruder3 and Gregory Welch2,3,4

1 Computational and Information Sciences, CCDC US Army Research Laboratory, Playa Vista, CA, United States, 2 College of Nursing, University of Central Florida, Orlando, FL, United States, 3 Institute for Simulation and Training, University of Central Florida, Orlando, FL, United States, 4 Department of Computer Science, University of Central Florida, Orlando, FL, United States

Recent times have seen increasing interest in conversational assistants (e.g., Amazon

Edited by: Alexa) designed to help users in their daily tasks. In military settings, it is critical to design Varun Dutt, assistants that are, simultaneously, helpful and able to minimize the user’s cognitive Indian Institute of Technology Mandi, load. Here, we show that embodiment plays a key role in achieving that goal. We India present an experiment where participants engaged in an augmented reality version Reviewed by: Ion Juvina, of the relatively well-known desert survival task. Participants were paired with a voice Wright State University, United States assistant, an embodied assistant, or no assistant. The assistants made suggestions Michail Maniadakis, Foundation for Research verbally throughout the task, whereas the embodied assistant further used gestures and Technology - Hellas (FORTH), and emotion to communicate with the user. Our results indicate that both assistant Greece conditions led to higher performance over the no assistant condition, but the embodied *Correspondence: assistant achieved this with less cognitive burden on the decision maker than the Celso M. de Melo [email protected]; voice assistant, which is a novel contribution. We discuss implications for the design [email protected] of intelligent collaborative systems for the warﬁghter.

Specialty section: Keywords: intelligent virtual assistant, collaboration, cognitive load, embodiment, augmented reality This article was submitted to Cognitive Science, a section of the journal INTRODUCTION Frontiers in Psychology Received: 22 April 2020 In the near future, humans will be increasingly expected to team up with artificially intelligent Accepted: 23 October 2020 (AI) non-human partners to accomplish organizational objectives (Davenport and Harris, 2005; Published: 17 November 2020 Bohannon, 2014). This vision is motivated by rapid progress in AI technology that supports a Citation: growing range of applications, such as self-driving vehicles, automation of mundane and dangerous de Melo CM, Kim K, Norouzi N, tasks, processing large amounts of data at superhuman speeds, sensing the environment in ways Bruder G and Welch G (2020) that humans cannot (e.g., infrared), and so on. This technology is even more relevant given Reducing Cognitive Load the increasing complexity and dynamism of modern workplaces and operating environments. and Improving Warfighter Problem Solving With Intelligent However, the vision for AI will only materialize if humans are able to successfully collaborate with Virtual Assistants. these non-human partners. This is a difficult challenge as humans, on the one hand, do not fully Front. Psychol. 11:554706. understand how AI works and, on the other hand, are often already overburdened by the task doi: 10.3389/fpsyg.2020.554706 (Lee and See, 2004; Hancock et al., 2011; Schaefer et al., 2016).

Frontiers in Psychology| www.frontiersin.org 1 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 2

de Melo et al. Reducing Cognitive Load With Assistants

This challenge is even more critical in military domains, where on users’ cognitive load from conversational assistants is still not warfighters are under increased (physical and cognitive) pressure well understood. and often face life and death situations (Kott and Alberts, 2017; A fundamental limitation of conversational assistants, TRADOC, 2018; Kott and Stump, 2019). Since the potential however, is their lack of embodiment and consequent costs of failure are higher, warfighters tend to be even more limited ability to communicate non-verbally. Non-verbal reluctant to trust and collaborate with AI (Gillis, 2017). For these communication plays an important role in regulating reasons, it is important to gradually increase exposure of AI social interaction (Boone and Buck, 2003). Expressions of technology to the warfighter through training that explains how emotion, additionally, serve important social functions such AI works, its strengths, but also its limitations. Complementary, as communicating one’s beliefs, desires, and intentions to it is fundamental that, during mission execution, AI is capable others (Frijda and Mesquita, 1994; Keltner and Lerner, 2010; of actively promoting trust and collaboration by communicating van Kleef et al., 2010; de Melo et al., 2014). The information naturally, effectively, and efficiently with the warfighter, while conveyed by non-verbal cues, therefore, can be very important minimizing the warfighter cognitive load. in building trust and promoting cooperation with humans Cognitive load is commonly defined as the difference (Bickmore and Cassell, 2001; Frank, 2004). Non-verbal between the cognitive demands of the task and the user’s communication in technology systems is typically achieved available cognitive resources (e.g., attention and memory) through robotic (Breazeal, 2003) or virtual agents (Gratch (Hart and Wickens, 1990; Nachreiner, 1995; Mackay, 2000). et al., 2002). Because these systems are embodied, they support Several factors have been identified that influence cognitive load non-verbal communication with users, including expression of but, broadly, it is possible to distinguish between exogenous emotion. Research shows that embodied conversational assistants factors (e.g., task difficulty) and endogenous factors (e.g., can have positive effects in human–machine interaction (Beale information processing capabilities for perceiving, planning, and and Creed, 2002; Beun et al., 2003; Kim et al., 2018b; Shamekhi decision making). Furthermore, individual factors, such as skill et al., 2018), including building rapport (Gratch et al., 2006) and experience, influence cognitive load. The importance of and promoting cooperation with users (de Melo et al., 2011; de controlling cognitive load for task performance has been studied Melo and Terada, 2019, 2020). In the context of problem-solving across several domains, with several studies suggesting that tasks, pedagogical agents have been shown to be able to improve increased cognitive load can undermine performance (Cinaz learning and task performance (Lester et al., 1997; Atkinson, et al., 2013; Dadashi et al., 2013; Vidulich and Tsang, 2014; 2002). These improvements are typically achieved through Fallahi et al., 2016). More recently, there has been increasing gestures that focus the user’s attention or through affective cues interest on the impact that technology has on worker’s cognitive serving specific pedagogical purposes, such as motivating the load. Much research has focused on the relationship between user (Schroeder and Adesope, 2014). Embodied conversational automation and user’s boredom and mental underload (Ryu assistants, therefore, hold the promise of having at least all and Myung, 2005), and its negative consequences on users’ the benefits in terms of task performance as (non-embodied) ability to react in a timely manner during real or simulated conversational assistants, while having minimal impact on the flight (Shively et al., 1987; Battiste and Bortolussi, 1988), user’s cognitive load. air combat (Bittner et al., 1989), air traffic control (Block With increased immersion afforded to the user (Dey et al., 2010), and so on. However, another concern is that et al., 2018; Kim et al., 2018a), augmented reality (AR) has technology may increase user’s cognitive load due to the the potential to further enhance collaboration with AI. AR change of routine processes, stress due to the transition, and technology supports superimposition of virtual entities alongside general lack of understanding of how the technology works the physical space in the user’s field of view. Therefore, first, (Mackay, 2000). interaction with AI systems can occur as the user is fully Intelligent virtual assistants offer a promising route to immersed in the task and, second, virtual interfaces—such promote collaboration between humans and AI, while controlling as embodied assistants—can be integrated seamlessly in the the impact on the user’s cognitive load (Gratch et al., 2002). workspace. Increased user influence in AR environments often Conversational assistants—i.e., intelligent virtual assistants that occurs through an increased sense of social presence (Blascovich can communicate through natural language—have, in particular, et al., 2002). Kim et al.(2018b, 2019) investigated the effects of an been experiencing considerable commercial success—e.g., Apple embodied conversational assistant on the sense of social presence Siri, Amazon Alexa, and Microsoft Cortana (Hoy, 2018). The and confidence in the system and found that both factors basic premise is that advances in natural language processing positively impacted users’ perception of the system’s ability to be technology (Hirschberg and Manning, 2015) enable more aware and influence the real world, when compared to a (non- natural open-ended conversation with machines, which can then embodied) conversational assistant. Furthermore, participants provide information or carry out users’ instructions. Verbal perceived increased trust and competence in embodied assistants communication is also socially richer than other forms of than the non-embodied counterparts. Wang et al.(2019) also communication like text or email, as it can convey pragmatic conducted a study investigating user preference for different and affective information (Kiesler et al., 1984). Most current types of embodied or non-embodied assistants in AR while commercial systems, though, only have limited ability to convey performing a visual search task together, and showed that this social richness through speech. Moreover, whereas natural participants preferred the miniature embodied assistant since the communication is expected to improve productivity, the impact small size made assistants “more approachable and relatable.”

Frontiers in Psychology| www.frontiersin.org 2 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 3

de Melo et al. Reducing Cognitive Load With Assistants

A telepresence study, in contrast, suggested that participants opportunity to test how persuasive a technology (e.g., a virtual were more influenced by an avatar, presumably representing assistant) is on the participant’s decision making; and, second, it another participant, that was the same size as a real person has a clearly defined optimal solution1, which allows comparison than by a miniature avatar (Walker et al., 2019); however, in of different technological solutions on task performance. this study, both types of avatar were located at a distance from the working environment and were not fully integrated in the Assistant Suggestions task space. Overall, this research, thus, suggests that embodied The assistants in the voice and embodied conditions were trying conversational assistants in AR can be especially persuasive. to help the participants make better decisions during the task Here, we present an experiment where participants engaged in by providing suggestions that could potentially improve the task a relatively well-known collaborative problem-solving task—the score. The system recognized where the items were currently desert survival task (Lafferty and Eady, 1974)—with an embodied located during the task and calculated the current survival score assistant, a voice-only assistant, and no assistant. The task is continuously. In this way, the assistants could determine the item implemented in an AR environment (Microsoft HoloLens). The to suggest to move, such that the participants could make the assistants attempt to persuade participants with information largest improvement in the survival score if they followed the pertinent to the task—e.g., “I heard the human body needs suggestion. The recommendation, thus, would always improve certain amount of salt for survival. Why don’t you move it the score, with respect to the optimal solution, if followed. There up a bit?” They also provide general positive reinforcement were both positive and negative suggestions for each survival (e.g., “You are doing great!”) and show appreciation when the item, according to whether the suggestion was to move the item participant follows the assistant’s suggestion (e.g., “Great! You up or down in the ranking. For example, the positive suggestion listened to my suggestion.”). Embodied assistants further smile for the flashlight was “The flashlight could be a quick and reliable when making recommendations and, to focus the participant’s night signaling device. Why don’t you move it up a bit?” and attention, move toward and point to the target of the suggestion. the negative suggestion was “I think the flashlight battery will Our main hypothesis is that the embodied assistant will lead to be gone very quickly, so it might not be as useful as you expect. increased task performance, when compared to the voice-only Why don’t you move it down a bit?” Table 1 shows the full list assistant (H1a); in turn, both assistants will lead to improved of positive and negative suggestions for all items. There were performance relative to the no assistant condition (H1b). We also three different prompt variations for both moving up and down look at participants’ subjective cognitive load and hypothesize suggestions, e.g., “I think it’s ok to move it up/down,” and “I guess that embodied assistants will lead to lower cognitive load, when you can move it up/down more.” The assistant could also make compared to the voice-only assistant (H2). Finally, for the stronger suggestions expressing that the item position should assistant conditions, we also look at measures of social presence be adjusted a lot. For example, “I think you should move it and social richness (Lombard et al., 2009) and hypothesize that up/down a lot,” “Why don’t you move it up/down a lot in the both will be perceived to be higher with embodied than voice- ranking?,” and “I’m sure you can move it up/down quite a lot.” only assistants (H3). A preliminary report of these experimental The assistants could make the same suggestions repeatedly if the results was also presented at the IEEE Conference on Virtual item was still the best option to improve the task score; however, Reality and 3D User Interfaces (IEEE VR) 2020 (Kim et al., 2020). if there was nothing to change for the score, no suggestion was provided. Participants received up to 10 suggestions from the assistant throughout the task. It is important to know that the EXPERIMENT assistants allowed the participants to decide whether they would follow the assistant’s suggestions or not; thus, if they wanted to, Methods they could ignore the suggestion. After following suggestions, Design and Task the assistants performed appreciation prompts, such as “Thank The experiment followed a within-participants design, with you for listening to my suggestion,” which could encourage each participant engaging in the desert survival task with no more compliance by participants for follow-up suggestions. The assistant (control), a voice-only assistant, and an embodied assistant also gave general acknowledgment comments, which assistant. The presentation order for the assistant conditions included some variations of simple assuring comments, such as was counterbalanced across participants to minimize ordering “Okay,” “Good,” or “You are doing great so far.” effects. Based on the original version of the desert survival task (Lafferty and Eady, 1974), participants were given the following Voice-Only Assistant instructions: “You are in the Sonoran Desert, after a plane crash. Like other commercial systems (e.g., Alexa or Cortana), the You know you are 70 miles away from the nearest known assistant had a female voice. The speech prompts included task habitation. You should prioritize the items on the table, by the instructions, general acknowledgments, and the survival item importance of the item for your survival until you arrive there.” suggestions. The assistant’s speech was also displayed in text as The participant is then asked to prioritize between 15 items such subtitles to the participant. Otherwise, the assistant had no visual as bottle of water, jackknife, mirror, pistol, etc. This task has often cue throughout the task. been used in human–machine interaction studies (Morkes et al., 1The optimal solution was based on the advice of Alonzo W. Pond, M.A., a survival 1998; Burgoon et al., 2000; Walker et al., 2019) for at least two expert who was a former chief of the Desert Branch of the Arctic, Desert, Tropic reasons: first, since the solution is not obvious, it introduces an Information Center of the Air University at Maxwell Air Force Base.

Frontiers in Psychology| www.frontiersin.org 3 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 4

de Melo et al. Reducing Cognitive Load With Assistants

TABLE 1 | Positive and negative suggestions for the items in the desert survival Embodied Assistant task. The embodied assistant was implemented as a female character, Item Positive suggestion Negative suggestion as shown in Figure 1. Building on prior work indicating a preference for miniature-sized characters (Wang et al., 2019), we Air map The air map might be helpful to The air map might not be made the assistant miniature-sized. This decision also meant that check where you are heading useful, since you already know and how much you should go the direction and the place the assistant could move around the task space without forcing further. where you have to go. participants to avert their gaze. The assistant was animated Book You might need to hunt for Well, I guess the book about with idle body motion and blinking and, when speaking, the food, so the book about edible edible animals is not so useful. I lip motion was synchronized with the speech. When making a animals seems useful. think the problem could be suggestion, the assistant would move next to the item in question dehydration, not starvation. and make a pointing gesture (Figures 1a–c). If the participant Coat Well, I guess the top coat could I guess the top coat might not be a good means, for keeping be useful, with hot weather in followed the suggestion, the assistant would also perform an the moisture on your skin, and the desert. acknowledgment gesture, such as a bow or putting the hand protecting you from hot and dry to the chest (Figures 1d–f). When making a suggestion or weather. acknowledging, the assistant would gaze at the participant and Compass The compass would guide you You can guess the direction, show one of its smiling expressions (Figures 1h,i); otherwise, the the direction where you are based on the Sun and the facial expression would be neutral (Figure 1g). heading to. Moon. The compass might be of little use. Compress Well, I think the compress kit The compass would guide you AR Implementation could be used as rope, or as a the direction where you are The AR environment for the desert survival task was further protection against heading to. implemented using Microsoft HoloLens technology and the dehydration and sunlight. Unity game engine. To complete the task, participants had to Flashlight The flashlight could be a quick I think the flashlight battery will place real image markers illustrating the 15 survival items in and reliable night signaling be gone very quickly, so it might device. not be as useful as you expect. the order of importance on a virtual board, while experiencing Jackknife The jackknife would be useful I’m not sure, but maybe the AR visual and auditory stimuli (Figure 2). The image markers for rigging the shelter, and for jackknife is not often used in were attached to physical foam bases so that the participants cutting up the cactus for the desert. could intuitively grab them and move them around. To initiate moisture. the task, participants first looked at the marker with a desert Mirror I think the mirror could be quite I’m not sure if the mirror is that image and put it on a start placeholder, which was virtually useful to communicate your important for your survival. displayed on the table through the head-mounted display. Once presence, by using the reflection of sunlight. the desert image marker was placed in the start placeholder, the Parachute The parachute might be useful I guess the parachute might be instruction and state boards virtually appeared with 15-item as a shelter, or a signaling burdensome, to bring all day, placeholders on the table, where participants could place the device, and it could also be and I’m not sure how you survival items in their chosen order. When the item was placed used to make shade. would use it. in one of the placeholders, the placeholder turned to blue with Pistol I guess the pistol could be used I guess there are not many a clicking sound effect and a virtual image corresponding to the as a sounding device, and it is dangerous creatures in the item image was shown on it. Participants could freely re-order good for protecting yourself desert, so the pistol might not from wild animals. be that important. the items and check the status of placed items via a state board Raincoat I think the plastic raincoat could It’s the desert. I’m not sure why while performing the task. After all the 15 items were placed be useful, for protecting your you would need raincoats. in the item placeholders, a finish placeholder was shown in AR body moisture, and it might next to the desert image marker, and the instruction guided the even be used to extract some participants to put the desert marker on the finish placeholder water, by the temperature differential. to complete the task. Once participants placed the desert marker Salt I heard the human body needs I guess the salt tablets would on the finish placeholder, the task was completed, showing a certain amount of salt for just require more body water to message that guided them to call the experimenter. The size of survival. get rid of the increased salinity. each marker was 10 cm × 10 cm × 1 cm. The PTC Vuforia Sunglasses The sunglasses would make I guess the sunglasses are not Engine2 was used for marker recognition. The realistic voice eyes comfortable from the that important. There should be for the assistants was pre-recorded using the Julie English voice intense sunlight. many other ways to protect from Vocalware’s text-to-speech3. The embodied assistant was your eyes. a custom 3D model created using Adobe Fuse4 and FaceGen5. Vodka I guess the vodka could be I think the vodka might not be a helpful for a fire, or as good item to bring in the Body animations were created using Unity’s assets and inverse temporary coolant for the body. desert. Alcohol would absorb water. 2 Water Bottle I think it would be good to drink The water might not be that https://developer.vuforia.com/ 3 the water, so you can remain important. I guess there could https://www.vocalware.com clear-headed, when important be ways to easily get water in 4https://www.adobe.com/products/fuse.html decisions have to be made. the desert, like cactus. 5https://facegen.com/

Frontiers in Psychology| www.frontiersin.org 4 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 5

de Melo et al. Reducing Cognitive Load With Assistants

FIGURE 1 | Embodied assistant gestures and facial expressions: (a–c) pointing gestures, (d–f) acknowledgment gestures, (g) neutral facial expression, (h) subtle smile, and (i) strong smile.

FIGURE 2 | Desert survival task and assistant conditions: (a) participant’s physical space, (b) participant’s AR view for the no assistant (control) condition, (c) voice-only assistant condition, and (d) embodied assistant condition.

Frontiers in Psychology| www.frontiersin.org 5 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 6

de Melo et al. Reducing Cognitive Load With Assistants

kinematics engine, as well as Mixamo6 animations. Please see the through the calibration procedure on the HoloLens to set their Supplementary Material for a video of the software. interpupillary distance. Afterward, participants had a practice session to learn how to use our marker-based interface by placing Measures five animal markers. In this practice phase, they were asked to The main measure was the score in the desert survival task. Score place the five animal markers in their preferred order on the table was calculated by summing the absolute differences between while experiencing AR visual feedback similar to the task. The the participant’s ranking and the optimal ranking. To make the experimenter was present next to the participants to answer any interpretation more intuitive, the sum of the differences was questions that they might have during the practice phase, while negated. This way, the best score was zero, which means all the explaining the way to place and re-order the items. Once they felt items in the participant’s ranking match the optimal solution. The comfortable with the marker-based interface, the experimenter task performance gets worse as the score moves further along described their actual task, the desert survival task, and the goal the negative scale. to prioritize the 15 items for their survival in a desert. In the To measure subjective cognitive load, we used the NASA description, participants were told that they were going to take Task Load Index (NASA-TLX) scale (Hart, 2006). Subjective part in the same task three times with some variations. Then, scales are often used to measure cognitive load as they tend the first session started with one of the experimental conditions: to be less intrusive than physiological scales (Meshkati et al., either the control, the voice-only, or the embodied condition 1992). Even though several other subjective scales have been as described above. After completing the task, the participants proposed (e.g., SWAT, Workload Profile), the NASA-TLX scale were guided to complete several questionnaires measuring their is still one of the most widely used and is sensitive to cognitive perception of the experience in the desert survival task with load, highly correlated with task performance, highly correlated or without assistant. When they were done answering the with other subjective scales, but may be outperformed by other questionnaires, the experimenter guided them to repeat the same scales in task discrimination (Rubio et al., 2004). The NASA-TLX task in the next condition. Once the participants completed consists of six questions, each corresponding to one dimension all three conditions, they answered further demographics and of the perceived workload. For each dimension (mental demand, prior experience questionnaires, assessing their familiarity and physical demand, temporal demand, performance, effort, and experience with AR and virtual assistant technology. The frustration), the participants provide a score on a scale, from participants were not informed about their performance on the “Very low” to “Very high,” consisting of 21 tick marks effectively survival task throughout the experiment. At the end, participants identifying 5% delimitations on a scale of 0 to 100%. Participants were provided with a monetary compensation ($15). The entire then provide weights for each of the six dimensions via a series of experiment took about an hour for each participant. binary choices to assess which dimensions were most important for the task; these weights are then factored into the final score by Sample multiplying them with the dimension scores. We recruited 37 participants from the University of Central Regarding social presence, we adopted the social presence sub- Florida population for the experiment. Thirty-six participants scale from the Temple Presence Inventory (TPI) questionnaire completed the entire experiment, while one withdrew for (Lombard et al., 2009) and slightly modified it to assess personal reasons. We further excluded two more participants participants’ sense of togetherness in the same space with the due to a failure to record data; thus, we had 34 participants assistant, and the quality of the communication/interaction (25 male and 9 female, ages 18 to 33, M = 21.90, SD = 4.10) between them. The scale consists of seven questions on a seven- for the analysis. All participants had normal or corrected-to- point scale (1, not at all, to 7, very much). We used this normal vision—12 with glasses and 7 with contact lenses. On questionnaire only for a subjective comparison of the assistant a seven-point scale (from 1, not familiar at all, to 7, very conditions, i.e., the voice-only vs. embodied assistant conditions. familiar), the level of participant-reported familiarity with AR For social richness, we adopted the social richness sub-scale from technology was comparatively high (M = 4.56, SD = 1.33). the TPI questionnaire (Lombard et al., 2009) to assess the extent All participants had fewer than 10 previous AR head-mounted to which the assistant is perceived as immediate, emotional, display experiences, and it was the first experience for 13 of responsive, lively, personal, sensitive, and sociable. All the items them. Participants were also asked about their frequency of using for social richness are seven-point semantic differential scales commercial conversational assistant systems, such as Amazon (e.g., 1, remote, to 7, immediate). We also used this questionnaire Alexa, Apple Siri, or Microsoft Cortana. Their responses varied only for the assistant conditions. from no use at all to frequent daily use: eight participants indicated multiple times per day, two indicated once a day, eight Procedure indicated once a couple of days, seven indicated once a week, Once participants arrived, they were guided to our laboratory three indicated once a month, and six indicated no use at all. Five space by an experimenter. They were asked to sit down in a participants had prior experience with the desert survival task or room with a table and a laptop PC for answering questionnaires closely related tasks. and were provided with the consent form. Once they agreed to participate in the experiment, they donned a HoloLens and went Results For task performance and subjective cognitive load, we ran a 6https://www.mixamo.com/ repeated-measures ANOVA and conducted post hoc comparisons

Frontiers in Psychology| www.frontiersin.org 6 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 7

de Melo et al. Reducing Cognitive Load With Assistants

with Bonferroni corrections. For cases where sphericity was not was worse than with the voice-only (p = 0.002) and embodied assumed, based on Mauchly’s test, we used the Greenhouse– assistants (p = 0.005); moreover, there was no statistically Geisser correction if the Greenhouse–Geisser epsilon was lower significant difference between the voice-only and embodied than 0.750 or, otherwise, the Huynh–Feldt correction. For social assistants (p = 1.000). The results suggest that assistants were presence and social richness, we conducted repeated measures able to improve participants’ performance, confirming hypothesis t tests to compare the voice-only vs. embodied assistant scores. H1b; however, we found no support that embodied assistants Figure 3 and Table 2 show the descriptive statistics for the improved performance when compared to voice-only assistants experimental results. (hypothesis H1a). Since we followed a within-participants design, we also wanted Task Performance to get insight into any possible order effects. Accordingly, we The participants’ scores in the desert survival task are shown in ran a repeated-measures ANOVA on the task scores for the Figure 3A. Recall that the optimal score is zero and that lower 1st, 2nd, and 3rd games. The results confirmed an order effect, 2 scores are worse. The analysis revealed a main effect for the F(1.405,66) = 16.029, p < 0.001, ηp = 0.327: participants got 2 assistant conditions, F(2,66) = 10.166, p < 0.001, ηp = 0.236. the lowest score, averaged across all assistant conditions, on Post hoc comparisons revealed that the score with no assistant the 1st game (M = −63.35, SD = 19.974), followed by the

FIGURE 3 | Task performance (A), cognitive load (B,C), social presence (D), and social richness (E) experimental results. The error bars correspond to standard errors. ∗p < 0.05.

Frontiers in Psychology| www.frontiersin.org 7 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 8

de Melo et al. Reducing Cognitive Load With Assistants

TABLE 2 | Means and standard errors for task performance, cognitive load, social presence, and social richness for all assistant conditions.

No assistant Voice assistant Embodied assistant

Mean SE Mean SE Mean SE

Performance −62.71 3.67 −50.88 3.65 −52.29 3.40 Subjective cognitive load 23.73 3.03 29.37 3.26 25.94 3.06 Mental demand 26.62 4.25 33.97 4.10 28.53 4.07 Physical demand 18.09 3.74 19.12 3.67 18.68 3.73 Temporal demand 11.18 2.70 12.21 2.76 13.68 2.50 Performance 27.79 6.33 31.18 6.11 30.44 6.43 Effort 21.91 3.74 28.09 3.94 24.12 3.75 Frustration 7.21 2.24 16.62 3.47 18.53 3.61 Social presence – – 3.59 0.20 4.60 0.19 Social richness – – 4.05 0.18 4.55 0.16

2nd game (M = −54.29, SD = 20.704), and, finally, the 3rd The results also showed an effect on social richness, t(33) = 2.565, game (M = −48.24, SD = 20.986). This result suggests that there p = 0.015, d = 0.408, with participants experiencing higher social were learning effects with participants becoming more proficient richness with embodied than voice-only assistants (Figure 3E). at solving the task with each attempt, independently of the Thus, the results confirmed our hypothesis H3. experienced assistant condition order. Nevertheless, the score pattern with the assistants in the 1st, 2nd, and 3rd games was the same as reported in the previous paragraph, though the effect was GENERAL DISCUSSION only statistically significant in the first game (1st game: p = 0.023, 2nd game: p = 0.168; 3rd game: p = 0.203). As AI technology becomes pervasive in the modern workplace, teams consisting of humans and machines will become Subjective Cognitive Load commonplace, but only if AI is able to successfully collaborate Following the established method, the NASA-TLX scores were with humans. This requirement is even more critical in calculated by summing the weighted sub-dimension scores (Hart, military domains, where warfighters are engaged in high-stakes, 2006). The overall task load results are shown in Figure 3B and complex, and dynamic environments and, thus, under immense the dimensions are shown in Figure 3C. The analysis revealed cognitive pressure (Kott and Alberts, 2017; Kott and Stump, a main effect for subjective cognitive load, F(1.631,66) = 5.243, 2019). Here, we present evidence suggesting that intelligent 2 p = 0.012, ηp = 0.137. Post hoc comparisons confirmed that virtual assistant technology can be a solution for improving the voice-only assistant led to higher load than no assistant human task performance, while controlling cognitive load. Our (p = 0.015) and embodied assistant conditions (p = 0.026). experimental results indicate that, in an abstract problem-solving There was no statistical difference between the no assistant and task, participants were able to produce higher-quality solutions embodied assistant conditions (p = 0.864). The result, thus, when partnered with assistants, when compared to no assistants. confirmed that embodied assistants led to lower cognitive load Unlike our initial expectations though, embodied assistants than voice-only assistants, in line with hypothesis H2. When did not improve performance over voice-only assistants. Prior looking at the NASA-TLX’s underlying dimensions, there was research suggests that the ability to communicate non-verbally— a trend for a main effect on mental demand—F(2,66) = 2.607, e.g., bodily postures and facial expressions—can lead to increased 2 p = 0.081, ηp = 0.073—and effort—F(2,66) = 2.020, p = 0.141, rapport (Gratch et al., 2006) and cooperation (de Melo et al., 2 ηp = 0.058—in line with the overall effect on cognitive load. In 2011, 2014; de Melo and Terada, 2019, 2020) with users and, contrast, there was a main effect on frustration, F(2,66) = 8.230, consequently, improved performance in collaborative tasks. In 2 p = 0.001, ηp = 0.200, with participants reporting increased this case, though, it seems that the information communicated frustration with the voice-only (p = 0.016) and embodied verbally had the most relevance for task performance, as assistants (p = 0.004) when compared to no assistant. This may suggested by some of the participants’ comments: have occurred because the recommendations were perceived as P4: “the assistant was very helpful, giving critical information in repetitive in some cases—for instance, one participant noted “the such a stressful situation if it happens in real world.” P7: “The person slightly annoyed me, she kept repeating the same advice interaction of the assistant was overall beneficial, as it brought up that I clearly did not care about”. many things I wouldn’t have thought of.” P12: “The information Social Presence and Social Richness provided by the assistant were great, it helped me prioritize items better.” Regarding social presence, the analysis revealed a statistically significant difference between the assistants, t(33) = 4.568, Interestingly, even though both assistants produced a bump p < 0.001, d = 0.622, with participants experiencing higher social in performance, the embodied assistant accomplished this with presence with embodied than voice-only assistants (Figure 3D). minimal impact on cognitive burden, as measured by the

Frontiers in Psychology| www.frontiersin.org 8 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 9

de Melo et al. Reducing Cognitive Load With Assistants

NASA-TLX scale, when compared to the voice-only assistant. may have played a role in reducing the participants’ subjective This is particularly relevant given the attention that voice- cognitive load while performing the task. only assistants—e.g., Amazon Alexa, Apple Siri, Microsoft The current work has limitations that introduce opportunities Cortana—have been receiving (Hoy, 2018). Prior work with for future work. First, we have only explored simple emotion embodied assistants, especially in pedagogical applications, has expression in the current work, with the assistant only showing been inconclusive about the impact of embodiment on cognitive various smiles throughout. However, research indicates that load (Schroeder and Adesope, 2014): in some cases, post-tests the social impact of emotion expressions depends on context were easier after interaction with an embodied agent (Moreno (Hess and Hareli, 2016), and even smiles can lead to reduced et al., 2001); in other cases, embodied agents led to increased cooperation when timed improperly (de Melo et al., 2014). cognitive load for learners, even though there was no impact Future work should consider more sophisticated emotion on performance (Choi and Clark, 2006). Mayer’s multimedia communication, which may lead to increased persuasiveness learning theory can provide insight here (Mayer, 2001). by the assistant (Shamekhi et al., 2018) and, ultimately, better Accordingly, there are two fundamental channels (auditory performance. Second, our current speech synthesizer—like most and visual) and optimal learning occurs when information is commercial systems—has limited expressive ability. However, as optimized across the channels—e.g., embodied agents would not speech technology improves, it will become possible to increase produce an effect if they are redundant or irrelevant to the the bandwidth of multimodal expression (Gratch et al., 2002). task (Craig et al., 2002). In our case, though, the embodied Optimized multimodal expression can, then, lead to optimized assistant was serving clear functions above and beyond the voice- transfer of information, learning, and performance (Mayer, only assistant: through facial expressions, it smiled when making 2008). Third, the current within-subjects design with a relatively suggestions; through its virtual body, it moved and pointed to small sample size could influence the participants’ performance the target of the suggestions; and, generally, through subtle cues and perception. Future work should complement the present (e.g., idle motion or blinking), the assistant conveyed a human- work with between-subject designs. Still, when we compared like presence in the task. The experimental results confirm that the participants’ first trials as between-subjects comparisons, we these kinds of non-verbal cues have meaningful impact on users’ found promising trends corresponding to our present results subjective cognitive load. Participants’ comments, such as the although not all the measures showed statistical significances, one below, support the notion of a benefit of visual embodiment which encourages us to consider a further investigation in a for helping participants feel more comfortable in collaborative between-subjects design with a large sample size. Fourth, we situations with virtual assistants: used Microsoft HoloLens in our experiment, but this was still the first generation of the prototype and, in practice, some P28: “I like that the assistant is naturally in front of you and given participants still complained about the weight and bulkiness of at the same time as I worked rather than pausing just to listen to what she had to say.” the device. As AR head-mounted displays become better (e.g., lighter and supporting wider field of views), we can expect The results also showed that participants experienced increased immersion and impact of embodied assistants. Sixth, higher social presence with embodied assistants than we measured subjective cognitive load using the NASA-TLX conversational assistants. Social presence relates to the ability of scale; however, these findings should be complemented with a communication medium to convey the sense that the user is physiological measures of cognitive load (Meshkati et al., 1992). immersed in the communication space and engaging in social Seventh, whereas the desert survival task captures many relevant interaction just as if it were face-to-face interaction (Short et al., aspects of collaborative problem solving, it is worth conducting 1976; Lowenthal, 2010). Research indicates that immersive further experimentation in more complex realistic tasks and technology—such as virtual or AR—have the potential to provide attempt to replicate the effects reported here. Finally, the current an increased sense of social presence, when compared to other prototype only implemented simple AI (e.g., to determine media, such as phone or desktop (Blascovich et al., 2002). optimal suggestions and whether the participant followed Our results, thus, support the idea that AR afforded increased suggestions), but it is possible to embed more intelligence and immersion in the interaction with the embodied assistant, autonomy into assistant technology; the higher the autonomy, the which may have contributed to reduced cognitive load. Related more important is non-verbal behavior likely to be (Gratch et al., to the social presence effect, our results further indicate that 2006; de Melo et al., 2011, 2014; de Melo and Terada, 2019). participants perceived higher social richness with the embodied Finally, the present work has important practical implications. than the voice-only assistant. This suggests that people were more The results confirm that assistant technology can improve task likely to treat interaction with the embodied assistant in a social performance, if deployed appropriately. Even with a voice- manner, as if they were interacting with another human. This is only assistant, we were able to show a clear improvement in line with prior research indicating that increased human-like in problem solving performance. Given the pace of evolution cues (Reeves and Nass, 1996) and immersion (Blascovich et al., of natural language processing technology (Hirschberg and 2002) can lead users to treat human–agent interaction like Manning, 2015), we can expect voice-only assistants to keep human–human interaction (Reeves and Nass, 1996; Mayer et al., playing a pervasive and influential role. However, the results 2003), which can lead to positive effects in terms of engagement, clearly indicate that voice-only assistants are fundamentally motivation, and interest (van Mulken et al., 1998; Atkinson, limited due to their inability to communicate non-verbally. 2002; Moreno, 2005; Schroeder and Adesope, 2014). The social Embodied assistants have the capability to engage users richness of the experience with the embodied assistant, thus, multimodally—like humans do (Gratch et al., 2002)—and

Frontiers in Psychology| www.frontiersin.org 9 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 10

de Melo et al. Reducing Cognitive Load With Assistants

complement the information conveyed through speech with AUTHOR CONTRIBUTIONS appropriate gestures and emotion. Given increasing evidence of the important role of non-verbal and emotional expression in CM and KK contributed to conception and design of the study, social interaction (Frijda and Mesquita, 1994; Boone and Buck, performed the statistical analysis, and wrote the first draft of 2003; Keltner and Lerner, 2010; van Kleef et al., 2010; de Melo the manuscript. KK and NN executed the study. All authors et al., 2014), developers and designers cannot afford to ignore contributed to manuscript revision, and read and approved the the value of embodiment for assistant technology. Our results submitted version. indicate that, by using non-verbal cues judiciously, we are able to control the users’ cognitive load while boosting performance, which is particularly critical when the stakes are high. This means that warfighters would be able to benefit from recommendations FUNDING and actions of the assistant technology, while being able to This material includes work supported in part by the National focus their cognitive resources on other aspects of the task. Science Foundation under Collaborative Award Numbers Alternatively, this would support even more communication 1800961, 1800947, and 1800922 (Dr. Ephraim P. Glinert, IIS) exchange and interaction with assistants, without overburdening to the University of Central Florida, University of Florida, and the warfighter. The fast pace of development in AI technology Stanford University, respectively; the Office of Naval Research and experimental research such as the one presented here clarifies under Award Number N00014-17-1-2927 (Dr. Peter Squire, how best to deploy this technology and introduces unique Code 34); and the AdventHealth Endowed Chair in Healthcare opportunities to create assistant technology that is immersive, Simulation (GW). Any opinions, findings, and conclusions feels like social interaction, is engaging and, most importantly, or recommendations expressed in this material are those of can promote optimal performance for the modern workforce and the author(s) and do not necessarily reflect the views of the the warfighter in increasingly complex operating environments. supporting institutions. This research was partially supported by the United States Army. The content does not necessarily reflect DATA AVAILABILITY STATEMENT the position or the policy of any Government, and no official endorsement should be inferred. All datasets generated for this study are included in the article/Supplementary Material. SUPPLEMENTARY MATERIAL ETHICS STATEMENT The Supplementary Material for this article can be found All experimental methods were approved by the UCF online at: https://www.frontiersin.org/articles/10.3389/fpsyg. Institutional Review Board (#FWA00000351 and IRB00001138). 2020.554706/full#supplementary-material

REFERENCES Block, R., Hancock, P., and Zakay, D. (2010). How cognitive load affects duration judgments: a meta-analytic review. Acta Psychol. 134, 330–343. doi: 10.1016/j. Atkinson, R. (2002). Optimizing learning from examples using animated actpsy.2010.03.006 pedagogical agents. J. Educ. Psychol. 94, 416–427. doi: 10.1037/0022-0663.94. Bohannon, J. (2014). Meet your new co-worker. Science 19, 180–181. doi: 10.1126/ 2.416 science.346.6206.180 Battiste, V., and Bortolussi, M. (1988). “Transport pilot workload: a comparison Boone, R., and Buck, R. (2003). Emotional expressivity and trustworthiness: the of two subjective techniques,” in Proceedings of the Human Factors Society role of nonverbal behavior in the evolution of cooperation. J. Nonverbal Behav. (Santa Monica, CA: Human Factors Society), 150–154. doi: 10.1177/ 27, 163–182. 154193128803200232 Breazeal, C. (2003). Toward sociable robots. Robot. Auton. Syst. 42, 167–175. doi: Beale, R., and Creed, C. (2002). Affective interaction: how emotional agents affect 10.1016/s0921-8890(02)00373-1 users. Int. J. Hum. Comp. St. 67, 755–776. doi: 10.1016/j.ijhcs.2009.05.001 Burgoon, J., Bonito, J., Bengtsson, B., Cederberg, C., Lundeberg, M., and Allspach, Beun, R.-J., De Vos, E., and Witteman, C. (2003). “Embodied conversational agents: L. (2000). Interactivity in human-computer interaction: a study of credibility, effects on memory performance and anthropomorphisation,” in Proceedings understanding, and influence. Comp. Hum. Behav. 16, 553–574. doi: 10.1016/ of the International Workshop on Intelligent Virtual Agents, Paris, 315–319. s0747-5632(00)00029-7 doi: 10.1007/978-3-540-39396-2_52 Choi, S., and Clark, R. (2006). Cognitive and affective benefits of Bickmore, T., and Cassell, J. (2001). “Relational agents: a model and an animated pedagogical agent for learning English as a second implementation of building user trust,” in Proceedings of the ACM SIGCHI language. J. Educ. Comp. Res. 24, 441–466. doi: 10.2190/a064-u776-420 Conference on Human Factors in Computing Systems, New York, NY, 396–403. 8-n145 Bittner, A., Byers, J., Hill, S., Zaklad, A., and Christ, R. (1989). “Generic workload Cinaz, B., Arnrich, B., Marca, R., and Tröster, G. (2013). Monitoring of mental ratings of a mobile air defence system (LOS-F-H),” in Proceedings of the Human workload levels during an everyday life office-work scenario. Pers. Ubiquitous Factors Society (Santa Monica, CA: Human Factors Society), 1476–1480. doi: Comput. 17, 229–239. doi: 10.1007/s00779-011-0466-1 10.1177/154193128903302026 Craig, S., Gholson, B., and Driscoll, D. (2002). Animated pedagogical agents Blascovich, J., Loomis, J., Beall, A., Swinth, K., Hoyt, C., and Bailenson, J. (2002). in multimedia educational environments: effects of agent properties, picture Immersive virtual environment technology as a methodological tool for social features, and redundancy. J. Educ. Psychol. 94, 428–434. doi: 10.1037/0022- psychology. Psychol. Inq. 13, 103–124. doi: 10.1207/s15327965pli1302_01 0663.94.2.428

Frontiers in Psychology| www.frontiersin.org 10 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 11

de Melo et al. Reducing Cognitive Load With Assistants

Dadashi, N., Stedmon, A., and Pridmore, T. (2013). Semi-automated CCTV Proceedings of the IEEE International Symposium on Mixed and Augmented surveillance: the effects of system confidence, system accuracy and task Reality (ISMAR), Washington, DC, 105–114. complexity on operator vigilance, reliance and workload. Appl. Ergon. 44, Kim, K., de Melo, C., Norouzi, N., Bruder, G., and Welch, G. (2020). “Reducing task 730–738. doi: 10.1016/j.apergo.2012.04.012 load with an embodied intelligent virtual assistant for improved performance in Davenport, T., and Harris, J. (2005). Automated decision making comes of age. collaborative decision making,”in Proceedings of the IEEE Conference on Virtual MIT Sloan Manag. Rev. 46, 83–89. Reality and 3D User Interfaces (IEEE VR), Reutlingen, 529–538. de Melo, C., Carnevale, P., and Gratch, J. (2011). The impact of emotion displays Kim, K., Norouzi, N., Losekamp, T., Bruder, G., Anderson, M., and Welch, G. in embodied agents on emergence of cooperation with people. Presence 20, (2019). “Effects of patient care assistant embodiment and computer mediation 449–465. doi: 10.1162/pres_a_00062 on user experience,” in Proceedings of IEEE International Conference on de Melo, C., Carnevale, P., Read, S., and Gratch, J. (2014). Reading people’s minds Artificial Intelligence and Virtual Reality, San Diego, CA, 17–24. from emotion expressions in interdependent decision making. J. Pers. Soc. Kott, A., and Alberts, D. (2017). How do you command an army of intelligent Psychol. 106, 73–88. doi: 10.1037/a0034251 things? Computer 50, 96–100. doi: 10.1109/mc.2017.4451205 de Melo, C., and Terada, K. (2019). Cooperation with autonomous machines Kott, A., and Stump, E. (2019). “Intelligent autonomous things on the battlefield,” through culture and emotion. PLoS One 14:e0224758. doi: 10.1371/journal. in Artificial Intelligence for the Internet of Everything, eds W. Lawless, R. Mittu, pone.0224758 D. Sofge, I. Moskowitz, and S. Russell (Cambridge, MA: Academic Press), de Melo, C., and Terada, K. (2020). The interplay of emotion expressions and 47–65. doi: 10.1016/b978-0-12-817636-8.00003-x strategy in promoting cooperation in the iterated prisoner’s dilemma. Sci. Rep. Lafferty, J., and Eady, P. (1974). The Desert Survival Problem. Plymouth, MI: 10:14959. Experimental Learning Methods. Dey, A., Billinghurst, M., Lindeman, R., and Swan, J. (2018). A systematic review Lee, D., and See, K. (2004). Trust in automation: designing for appropriate reliance. of 10 years of augmented reality usability studies: 2005 to 2014. Front. Robot. AI Hum. Factors 46, 50–80. doi: 10.1518/hfes.46.1.50.30392 5:37. doi: 10.3389/frobt.2018.00037 Lester, J., Converse, S., Kahler, S., Barlow, S., Stone, B., and Bhogal, R. (1997). “The Fallahi, M., Motamedzade, M., Heidarimoghadam, R., Soltanian, A., and Miyake, persona effect: affective impact of animated pedagogical agents,” in Proceedings S. (2016). Effects of mental workload on physiological and subjective responses of the ACM SIGCHI Conference on Human Factors in Computing Systems, during traffic density monitoring: a field study. Appl. Ergon. 52, 95–103. doi: Reutlingen, 359–366. 10.1016/j.apergo.2015.07.009 Lombard, M., Ditton, T., and Weinstein, L. (2009). “Measuring presence: the Frank, R. (2004). “Introducing moral emotions into models of rational choice,” in Temple presence inventory,” in Proceedings of the International Workshop on Feelings and Emotions, eds A. Manstead, N. Frijda, and A. Fischer (New York, Presence, Los Angeles, CA, 1–15. NY: Cambridge University Press), 422–440. doi: 10.1017/cbo9780511806 Lowenthal, P. (2010). “Social presence,” in Social Computing: Concepts, 582.024 Methodologies, Tools, and Applications, ed. S. Dasgupta (Hershey, PA: IGI Frijda, N., and Mesquita, B. (1994). “The social roles and functions of emotions,” Global), 202–211. in Emotion and Culture: Empirical Studies of Mutual Influence, eds S. Kitayama, Mackay, W. (2000). Responding to cognitive overload: co-adaptation between and H. Markus (Washington, DC: American Psychological Association). users and technology. Intellectica 30, 177–193. doi: 10.3406/intel.2000.1597 Gillis, J. (2017). Warfighter trust in autonomy. DSIAC J. 4, 23–29. Mayer, R. (2001). Multimedia Learning. New York, NY: Cambridge University Gratch, J., Okhmatovskaia, A., Lamothe, F., Marsella, S., and Morales, M. (2006). Press. “Virtual rapport,” in Proceedings of the International Conference on Intelligent Mayer, R. (2008). Applying the science of learning: evidence-based principles for Virtual Agents (IVA), Los Angeles, CA, 14–27. doi: 10.1007/11821830_2 the design of multimedia instruction. Am. Psychol. 63, 760–760. doi: 10.1037/ Gratch, J., Rickel, J., André, E., Cassell, J., Petajan, E., and Badler, N. (2002). 0003-066X.63.8.760 Creating interactive virtual humans: some assembly required. IEEE Intell. Syst. Mayer, R., Sabko, K., and Mautone, P. (2003). Social cues in multimedia learning: 17, 54–63. doi: 10.1109/mis.2002.1024753 role of speaker’s voice. J. Educ. Psychol. 95, 419–425. doi: 10.1037/0022-0663. Hancock, P., Billings, D., Schaefer, K., Chen, J., de Visser, E., and Parasuraman, R. 95.2.419 (2011). A meta-analysis of factors affecting trust in human-robot interaction. Meshkati, N., Hancock, P., and Rahimi, M. (1992). “Techniques in mental Hum. Factors 53, 517–527. doi: 10.1177/0018720811417254 workload assessment,” in Evaluation of Human Work. A Practical Ergonomics Hart, S. (2006). “NASA-task load index (NASA-TLX); 20 years later,”in Proceedings Methodology, eds J. Wilson, and E. Corlett (London: Taylor & Francis), 605–627. of the Human Factors and Ergonomics Society, Thousand Oaks, CA, 904–908. Moreno, R. (2005). “Multimedia learning with animated pedagogical agents,” in doi: 10.1177/154193120605000909 The Cambridge Handbook of Multimedia Learning, ed. R. Mayer (Cambridge: Hart, S., and Wickens, C. (1990). “Workload assessment and prediction,” in Cambridge University Press), 507–524. doi: 10.1017/cbo9780511816819.032 Manprint, ed. H. Booher (Springer), 257–296. doi: 10.1007/978-94-009- Moreno, R., Mayer, R., Spires, H., and Lester, J. (2001). The case for social agency 0437-8_9 in computer-based teaching: do students learn more deeply when they interact Hess, U., and Hareli, S. (2016). “The impact of context on the perception of with animated pedagogical agents? Cogn. Instr. 521, 177–213. doi: 10.1207/ emotions,” in The Expression of Emotion: Philosophical, Psychological, and Legal s1532690xci1902_02 Perspectives, eds C. Abell, and J. Smith (Cambridge: Cambridge University Morkes, J., Kernal, H., and Nass, C. (1998). “Humor in task-oriented computer- Press). mediated communication and human-computer interaction,” in Proceedings of Hirschberg, J., and Manning, C. (2015). Advances in natural language processing. ACM SIGCHI Conference on Human Factors in Computing Systems, New York, Science 17, 261–266. NY, 215–216. Hoy, M. (2018). Alexa, Siri, Cortana, and more: an introduction to voice assistants. Nachreiner, F. (1995). Standards for ergonomics principles relating to the design of Med. Ref. Serv. Q. 37, 81–88. doi: 10.1080/02763869.2018.1404391 work systems and to mental workload. Appl. Ergon. 26, 259–263. doi: 10.1016/ Keltner, D., and Lerner, J. (2010). “Emotion,”in The Handbook of Social Psychology, 0003-6870(95)00029-c eds D. Gilbert, S. Fiske, and G. Lindzey (John Wiley & Sons), 312–347. Reeves, B., and Nass, C. (1996). The Media Equation: How People Treat Computers, Kiesler, S., Siegel, J., and McGuire, T. (1984). Social psychological aspects of Television, and New Media Like Real People and Places. New York, NY: computer-mediated communication. Am. Psychol 39, 1123–1134. doi: 10.1037/ Cambridge University Press. 0003-066x.39.10.1123 Rubio, S., Díaz, E., Martín, J., and Puente, J. (2004). Evaluation of subjective Kim, K., Billinghurst, M., Bruder, G., Duh, H., and Welch, G. (2018a). Revisiting mental workload: a comparison of SWAT, NASA-TLX, and workload trends in augmented reality research: a review of the 2nd decade of ISMAR profile methods. App. Psychol. 53, 61–86. doi: 10.1111/j.1464-0597.2004. (2008-2017). IEEE Trans. Vis. Comp. Gr. 24, 2947–2962. doi: 10.1109/tvcg.2018. 00161.x 2868591 Ryu, K., and Myung, R. (2005). Evaluation of mental workload with a combined Kim, K., Boelling, L., Haesler, S., Bailenson, J., Bruder, G., and Welch, G. (2018b). measure based on physiological indices during a dual task of tracking and “Does a digital assistant need a body? The influence of visual embodiment mental arithmetic. Int. J. Ind. Ergon. 35, 991–1009. doi: 10.1016/j.ergon.2005. and social behavior on the perception of intelligent virtual agents in AR,” in 04.005

Frontiers in Psychology| www.frontiersin.org 11 November 2020| Volume 11| Article 554706 fpsyg-11-554706 November 11, 2020 Time: 15:25 # 12

de Melo et al. Reducing Cognitive Load With Assistants

Schaefer, K., Chen, J., Szalma, J., and Hancock, P. (2016). A meta-analysis of van Mulken, S., Andre, E., and Muller, J. (1998). “The persona effect: how factors influencing the development of trust in automation: implications for substantial is it?,” in People and Computers XIII, eds H. Johnson, L. Nigay, and understanding autonomy in future systems. Hum. Factors 58, 377–400. doi: C. Roast (London: Springer), 53–66. doi: 10.1007/978-1-4471-3605-7_4 10.1177/0018720816634228 Vidulich, M., and Tsang, P. (2014). The confluence of situation awareness and Schroeder, N., and Adesope, O. (2014). A systematic review of pedagogical agents’ mental workload for adaptable human-machine systems. J. Cogn. Eng. Decis. persona, motivation, and cognitive load implications for learners. J. Res. Tech. Mak. 9, 95–97. doi: 10.1177/1555343414554805 Educ. 46, 229–251. doi: 10.1080/15391523.2014.888265 Walker, M., Szafir, D., and Rae, I. (2019). “The influence of size in augmented reality Shamekhi, A., Liao, Q., Wang, D., Bellamy, R., and Erickson, T. (2018). “Face telepresence avatars,” in Proceedings of the IEEE Conference on Virtual Reality value? Exploring the effects of embodiment for a group facilitation agent,” and 3D User Interfaces, Reutlingen, 538–546. in Proceedings of the 2018 CHI Conference on Human Factors in Computing Wang, I., Smith, J., and Ruiz, J. (2019). “Exploring virtual agents for augmented Systems, Montreal, QC. reality,” in Proceedings of the ACM SIGCHI Conference on Human Factors in Shively, R., Battiste, V., Matsumoto, J., Pepiton, D., Bortolussi, M., and Hart, S. Computing Systems, New York, NY, 1–12. (1987). “In flight evaluation of pilot workload measures for rotorcraft research,” in Proceedings of the Symposium on Aviation Psychology (Columbus, OH: Conflict of Interest: The authors declare that the research was conducted in the Department of Aviation, Ohio State University), 637–643. absence of any commercial or financial relationships that could be construed as a Short, J., Williams, E., and Christie, B. (1976). The Social Psychology of potential conflict of interest. Telecommunications. London: Wiley. TRADOC (2018). The U.S. Army in Multi-Domain Operations 2028. Minna: Copyright © 2020 de Melo, Kim, Norouzi, Bruder and Welch. This is an open-access TRADOC. article distributed under the terms of the Creative Commons Attribution License van Kleef, G., De Dreu, C., and Manstead, A. (2010). An interpersonal approach (CC BY). The use, distribution or reproduction in other forums is permitted, provided to emotion in social decision making: the emotions as social information the original author(s) and the copyright owner(s) are credited and that the original model. Adv. Exp. Soc. Psychol. 42, 45–96. doi: 10.1016/s0065-2601(10) publication in this journal is cited, in accordance with accepted academic practice. No 42002-x use, distribution or reproduction is permitted which does not comply with these terms.

Frontiers in Psychology| www.frontiersin.org 12 November 2020| Volume 11| Article 554706