Representation of real-world event schemas during narrative perception 360.28 Christopher Baldassano, Uri Hasson, Kenneth Norman

Summary Schema-specific Event Segmentation Model Data-driven Alignment Understanding and remembering everyday experiences Model latent event structure of narrative-driven brain activity Unsupervised event alignments on odd and even runs requires maintaining situation models of ongoing events Representations Shared PCC Shared PCC Hand-labeled events Average patterns (N-1 subjects) event patterns event patterns Situation models are built from schematic templates 1 2 3 4 1 2 3 4 Restaurant stories Within Restaurant stories (Odd runs) (Even runs) learned over a lifetime Bower, Black, Turner 1979 Across

Voxels Measure Full details: Baldassano et al. bioRxiv 2016, 10.1101/081018 event Ways to fit the model: Airport stories Airport stories (Odd runs) (Even runs) match Restaurant (1) Given event voxel patterns, find matching events Events Events 1 2 3 1 2 3 p=0.03 Voxel 1 Average patterns (1 subject) Voxel 1 Even runs

Voxel 2 0.7 1 Odd runs Voxel 2 Event pattern 2 Voxel 3 Voxel 3 correlation Compare within-schema 3 Time Events -0.7 0 0.9 event match to null 0.5 4 (2) Given timecourse, segment into stable events 1 2 3 4

Airport model (permutation) Events Events Using fMRI data from subjects watching and listening to 1 2 3 Voxel 1 Voxel 1

stories that share schemas, we find: (r) match Event 0.3 Voxel 2 Voxel 2

0 1 2 3 Voxel 3 Voxel 3 • High-level brain regions (including posterior cingulate Time (minutes) 0.1 Time Time cortex) have representations of schemas that Within Across (3) Given multiple timecourses, align matching events generalize across subjects, stories, and modalities 0.09 Posterior cingulate Even without stimulus labels, PCC can identify shared temporal structure among stories in a schema • Event transitions in a held-out story can be predicted Stimulus A 0.07 Stimulus A from PCC activity

0.05 • Unsupervised temporal clustering of PCC activity Next Step: Stimulus B separates stories from different schemas Stimulus B 0.03 Impact of Schemas on Memory Event match (r) match Event Subjects later freely recalled all stories in the scanner 0.01 Within schema Across modality Transferring Event Labels Examples of errors: only "Robert Downey Jr. gets on the plane eventually. First he notices Hand-labeled events Predicting event transitions Avg PCC event patterns Stimuli from 7 stories on held-out story this woman, he’s kind of looking around at the airport and he notices a woman that wants to get a window seat—WAIT wrong movie." Event 1 Event 2 Event 3 Event 4 "Yeah I don’t really remember much about what happened here with Enter Seated at Ordering Food the . Yeah it was sort of like a conversation between the two Restaurant: characters... um, yeah I think they were at a restaurant? But I’m Restaurant Table Food Arrives Time not actually sure." 100% Enter Boarding Finding "Uh Shame…I I uh…I don’t remember which one that was sorry." Airport: Security Airport Gate Seat p=0.005 Predictions Video Audio Correct recall requires both the story schema and The Big Bang Theory 50% story-specific details to be encoded and retrieved The Santa Clause 31 subjects Shame Using neural measures of encoding and reinstatement, predict: My Cousin Vinny Restaurant 2mm voxels

Timepoint classification accuracy Story Schema Story Details Behavior 1.5 sec TR Transfer accuracy Friends 0% Scrambled-event null Within schema Across modality q<0.05 Restaurant stories Airport stories Average Present Present Coherent, correct recall How I Met Your Mother 3 minute stories Present Absent Within-schema event confusion Seinfeld 48 minutes total Posterior cingulate, anterior temporal, and lateral frontal Schema event structure can be predicted in Airport Absent Present Incoherent recall Up in the Air regions have representations of schemas that unlabeled stimuli using PCC activity patterns generalize across stories and modalities Absent Absent No meaningful recall