Structured Text Analysis for Evaluating Shared Cognition Leslie DeChurch, Michael Schultz, Jeffrey Johnson, Noshir Contractor, Jessica Mesmer-Magnus, Gabriel Plummer, and Marlon Twyman

This research was supported by a grant from NASA (NNX15AM26G). Project SCALE: Shared Cognitive Architecture for Long Term Exploration When asked what problems he encountered most frequently while spending 520 days in a 550 cubic meter isolated module to simulate a mission to Mars, Diego Urbina reported “mind reading”… Some issues may have included misunderstandings between crew and ground due to the lack of live communications or culture. We had to try to read each others mind …we were also mindreading with the crew members speaking Russian, but you can communicate more easily with them than you can with the ground, one or more orders of magnitude easier than communication with ground when on a delay (DeChurch & Mesmer-Magnus, 2015). Shared Cognition

• Shared cognition within teams and work groups has been linked to stability, efficiency, performance, responses to stress (DeChurch & Mesmer-Magnus 2010; Salas et al. 2008; Salas & Cannon-Bowers 2000; Driskell & Salas 1991)

• Long distance space exploration will require shared cognition among crew members and between crew and ground control (Landon, Vessey, & Barrett, 2016)

• Shared cognition between teams positively relates to inter-team coordination & multiteam performance (DeChurch, 2002; Murase, Carter, DeChurch, Marks, 2014) Shared Cognition in Multiteam Systems

Crew is one team working in a larger system of teams Shared Cognition in Multiteam Systems

Crew is one team working in a larger system of teams

“Ties” are shared cognition Shared Cognition in Multiteam Systems

Crew Shared Cognition Crew is one team working in a larger system of teams

“Ties” are shared cognition Shared Cognition in Multiteam Systems

Crew-MCC Crew Shared Shared Cog. Cognition Crew is one team working in a larger system of teams

“Ties” are shared cognition Project SCALE

• Shared cognition has been elicited using surveys – not “scalable”

• We develop a text-based technique to measure and monitor the similarity of mental models within and across teams

• This technique is effective for characterizing relationships and diagnosing conflicts between crew and ground control for the three manned Skylab missions. Measuring Shared Cognition with Text

Conventional measurement of mental models requires elaborate survey instruments (Cooke et al. 2004): • Time-consuming; survey fatigue • Intrusive; potential response bias • Not “realtime” nor continuous

Diaries (auto-biographical) versus Conversational Analysis

Conversation-based measures: • Unintrusive, do not require attention, and can be run continually • Useful for analysis of cognition, interactions, and discourse (Evans &Aceves 2016) Network Model of Shared Cognition

Corpus Mental Models Lorem ipsum dolor sit amet, consectetur Topics adipiscingLorem ipsumelit. dolor Topics sit amet, consectetur Quisque quisLorem purus ipsum dolor magna.adipiscing Mauris elit. Quisquesit amet quis, consectetur purus metus dolor,adipiscing elit. venenatismagna.sit amet Mauris metusQuisque dolor, quis purus quam quismagna., porta Mauris sempervenenatis diam.sit amet quam quismetus, porta dolor, Sentiment Sentiment Phasellusvenenatiset metus sit amet accumsansemper, tristique diam. Phasellusquamet quis metus, porta dui volutpatsemper, mollis diam. accumsanest. , tristique dui volutpatPhasellus, molliset metus accumsanest. , tristique dui volutpat, mollis est. Actors Shared Degree to which Cognition Group-level Aggregates the “topics- sentiment” in 2 Crew-MCC people’s utterances are correlated Actors Crew Sentiment Analysis

• We use the LIWC lexicon to measure sentiment within transcripts (Tausczik and Pennebaker 2010) • Wide range of lexical categories • Frequently used and well validated • Common in automated text analysis

• Focusing on: • cognitive processes (“cogmech”) • affect (“affect”), positive and negative Topic Modeling

• Used Latent Dirichlet Allocation to model topics (Blei 2012) • Topics are relatively coherent, interpretable, stable • Similar to topics produced by expert coders (Chang et al. 2009) • Modeled 20 topics over the corpus of tapes • Projected each utterance into topic space to create a topic vector for each speaker Case Study: Skylab Skylab Crews Three manned missions with three crewmembers: • Commander (CDR), Pilot (PLT), Scientist pilot (SPT) Skylab as an MTS Analog

• Mirrors ICC (Isolated, Confined, Controlled) conditions …

AND • Conceptually - Astronauts & CC experienced the high uncertainty of success inherent in deep space exploration missions • First time attempt to live and work in space for extended period of time

• Methodologically – Transcripts of “within” and “between” communication available Mission Details: Skylab 2

• Duration: 28 days • Working in space, Solar observations • 3 EVAs (one for docking) • Deployment of solar parasol • Technical difficulties and high involvement with mission control Mission Details: Skylab 3

• Duration: 59 days • Biological experiments, health research • 3 EVAs • Lost thruster, potentially mission threatening • “Low” involvement with mission control Mission Details: Skylab 4

• Duration: 84 days • Comet and solar observations • 4 EVAs • Space sickness hidden from ground control • Complaints about busy work schedule • Tension between mission control and crew • “Mutiny in space” Mission Details: Skylab 4

3 Explanations for conflict: • Mission control’s unrealistic expectations of workload (HBR) • Lack of experience in flight crew • Lack of crew-MCC joint training (Crew report) Data - Transcripts

• 2 channels: Air to ground communications & onboard voice transcription • ~15,000 pages of spoken communication, ~3,800 tapes • Identify time, speaker, and verbatim utterance • Trimmed to four most prevalent speakers: CDR, PLT, SPT, CC • Timing for unmarked utterances are interpolated using word count • Measurements converted into averages at each hour Results Topics Identified in LDA

Topic Label Words Topic Label Words Comet comet Maneuvering thrust perihelion pitch, yaw Comm legible Navigation horizon chat, howdy sextant, procyon Sun coma Repair cutter ultraviolet, glow foil, meteroid Instruments scatterometer Capsule deorbit radiometer, malfunction service, evaporation Solar Observation raster Contacts station aperture, sunspot places, names Experiments striation Repair/EVA visor seed tether, EVA Hygiene washcloth Consumption afrin spoon, trash biscuit, whiskey Politics Nixon Organization stowage Kissinger, congress retracted, opened Schedule printer Zero-G freedom pre-sleep, snack sensation, tumble Earth Observation intervalometer Miscellaneous Boston, airfield 1. More “cognitive” than “affective” 2. Little fluctuation in the Sentiment Analysis “amount” of either Cognitive Lexicon: Affective Lexicon: 737 Words 929 Words e.g. change, e.g. annoy, decide, idea chuckle, fantastic

Mission Day Skylab 2 Shared Cognition

More similar “within” than “between” Skylab 3 Shared Cognition

More similar “within” than “between” Skylab 4 Shared Cognition 1. Sometimes more similar “between” than “within” Skylab 4 Shared Cognition 1. Sometimes more similar “between” than “within” Skylab 4 Shared Cognition

2. Within & Between Similarity are not differentiated Shared Cognition – Day 7

Day 7

Crew is one team working in a larger Day 58 system of teams Shared Cognition – Day 58

Day 7

Skylab 4 – Intra-crew less shared cognition Day 58 Shared Cognition – Day 58

Day 7

Day 58

2. Skylab 4 - Crew-CC less shared cognition Shared Cognition State Space Shared Cognition State Space

Low Intra-crew High Intra-crew High Crew-CC High Crew-CC

Low Intra-crew High Intra-crew Low Crew-CC Low Crew-CC Skylab 2 & 3 Shared Cognition

Weighting by cognition lexicon Skylab 2 Shared Cognition Trajectory

Weighting by cognition lexicon Skylab 2 Stylized Shared Cognition Trajectory

Weighting by cognition lexicon Skylab 3 Shared Cognition Trajectory

Weighting by cognition lexicon Shared Cognition State Space

Weighting by cognition lexicon Shared Cognition State Space

Weighting by cognition lexicon Shared Cognition State Space

Weighting by cognition Day 6-8 lexicon

Day 56-60 Shared Cognition State Space

Weighting by affect lexicon Skylab 4 Shared Cognition Trajectory

Weighting by affect lexicon Skylab 4 Stylized Shared Cognition Trajectory

Weighting by Day 6-8 affect lexicon

Day 56-60 Shared Cognition Differences

Launch Mid-mission Return Skylab 2 Technical Issues High Intra-crew High Intra-crew High Intra-crew No People Issues Med Crew-CC High Crew-CC High Crew-CC

Skylab 3 Minor Technical Issues High Intra-crew High Intra-crew High Intra-crew No People Issues Low Crew-CC Low Crew-CC High Crew-CC

Skylab 4 Minor Technical Issues Low Intra-crew Med Intra-crew High Intra-crew High People Issues Low Crew-CC Med Crew-CC High Crew-CC Shared Cognition Differences

Launch Mid-mission Return Skylab 2 Technical Issues High Intra-crew High Intra-crew High Intra-crew No People Issues Med Crew-CC High Crew-CC High Crew-CC

Skylab 3 Minor Technical Issues High Intra-crew High Intra-crew High Intra-crew No People Issues Low Crew-CC Low Crew-CC High Crew-CC

Skylab 4 Minor Technical Issues Low Intra-crew Mod Intra-crew High Intra-crew High People Issues Low Crew-CC Mod Crew-CC High Crew-CC Conclusions Takeaways

• Alignment between task interdependence and shared cognition

• Importance of examining: • Location of similarity • Dynamics, shifts in similarity • Functionality of shifts relative to task interdependence Caveat

• Did people problems cause closer “inter” shared cognition, or did closer “inter” shared cognition cause people problems?

• Case studies do not afford causal inferences; Experiments in Project RED Project RED Multiteam Performance Metrics

Planetary Geology: Space Human available water at a Factors: total cost of given location training, research

Extraterrestrial Space Robotics: Engineering: total cost of the total water output of construction; labor the well efficiency

Project RED MTS: Time to signoff; # of times a signoff was attempted; # of Missing Variables; overall MTS success (the extent to which all teams succeeded)

Next Steps

• Continue to validate on other historical missions (Apollo, Gemini) • Test experimentally with Project RED (multi- team task platform) and the cooperation of HERA • Automate the process of monitoring and diagnosis Thank you! Takeaways (cont.)

• Skylab 4 had distinctly different patterns of mental model similarity • Differences within (lower) and between (intermediate) teams • Higher crew-CC similarity even during routine periods • Lower intra-crew similarity during most periods • Detectable early, and became more exaggerated late in the mission