Stories in GeoTime Ryan Eccles, Thomas Kapler, Robert Harper, William Wright Information Visualization ( 2008 ) © Stefan John 2008 Summer term 2008 1 Introduction • Story a powerful abstraction − Used by intelligence analysts to conceptualize threats and understand patterns − Part of the analytical process • Oculus Info’s GeoTime™ − Geo-temporal event visualization tool − Augmented with story system − Narratives, hypertext-linked visualizations, visual annotations, and pattern detection − Environment for analytic exploration and communication − Detects geo-temporal patterns − Integrates story narration to increase analytic sense-making cohesion Summer term 2008 2 1 Introduction • Assisting the analyst in: − Identifying, − Extracting, − Arranging, and − Presenting stories within the data • Story system − Lets analysts operate at story level − Higher level abstractions of data (behaviors and events) − Staying connected to the evidence − Developed in collaboration with analysts • Formal evaluation showed high utility and usability Summer term 2008 3 Overview • Storytelling • Related Work • Geo-Temporal Visualization in GeoTime • Stories in GeoTime • Evaluation Summer term 2008 4 2 Storytelling • First described in Aristotle’s Poetics − Objects of a tragedy (story): - Plot -> arrangements of incidents -Character - Thought -> processes of reasoning leading characters to their respective behavior • Narrative theory suggests: − People are essentially storytellers − Implicit ability to evaluate a story for: - Consistency -Detail - Structure Summer term 2008 5 Storytelling • Story offers context − Understand activities and plots played by characters or actors − Better comprehend the point being communicated by author − Helps audience to apply their tacit knowledge • Knowledge distinction − Tacit knowledge = knowledge carried by people in their minds - Difficult to access and to communicate - Not aware of − Explicit knowledge = knowledge that is easy to communicate − Process of transforming -> Articulation Summer term 2008 6 3 Storytelling • From analyst’s perspective: − Story offers a common form of communication − Investigating the feasibility of a connected collection of characters and their motives • Potent way of capturing the analysts’ insights − Promote sharing of observations − Facilitate understanding of complex phenomenon • Story as a mental framework -> Allows to organize daily observations into meaningful knowledge Summer term 2008 7 Related Work • Famous aphorism (M. Polanyi): “We know more than we can tell.” • Various approaches to connect information visualization with communicative power of storytelling • Investigated in context of usage by intelligence analysts • Common technique: Hypertext links to multimedia or additional textual content Summer term 2008 8 4 LifeLines Project • Allowed for incorporation of multimedia and interactions • Creation and playback of connected temporal data • Interactively linked 2D displays LifeLines – Visualizing Personal Histories Plaisent C., Milash B., Rose A. Widoff S, Shneiderman B. Proceedings of CHI ( 1996 ) Summer term 2008 9 MyLifeBits System • Incorporates lifetime’s media to tell a story MyLifeBits – Fulfilling the Memex Vision Gemmell J., Bell G., Lueder R., Drucker S., Wong C. ACM Multimedia ( 2002 ) Summer term 2008 10 5 Textable Movie System • Video content associated with text • Assemble pre-recorded video clips together to form a movie • Add narrative structure to existing content • Intended to be viewed, not manipulated or investigated further A System to Compose Movies for Cross-Cultural Story-Telling – Textable Movie Vaucelle C., Davenport G. Summer term 2008 Proceedings of TIDSE ( 2004 ) 11 Sense.us • Exploration of social aspects of collaborative analysis • Combined with annotation on visualization • Uses blog-style discussion workflow • Basis for communicating analysis • Annotations implemented as vector graphics overlaid above 2D visualization Voyagers and Voyeurs – Supporting Asynchronous Collaborative Information Visualization Heer J., Viegas F.B., Wattenberg M. CHI Proceedings ( 2007 ) Summer term 2008 12 6 Sense.us Summer term 2008 13 GeoTime • Time-space visualization framework • Unified temporal and geospatial analysis • Designed to improve perception and understanding of: − Entity movements, − Events, − Relationships, and − Interactions over time within a geospatial context GeoTime Information Visualization Kapler T., Wright W. IEEE InfoViz ( 2004 ) Summer term 2008 14 7 Geo-Temporal Representation • Events represented within an X, Y, T coordinate space − X and Y plane represents geographic space − Z-axis represents temporal space Summer term 2008 15 GeoTime • Enables analysis of information connectedness over time and geography within a single, highly interactive 3D view Summer term 2008 16 8 Velocity Annotation • Movement speed function − Annotates geo-temporal path with visual indicator of velocity − Red is fast / white is slow Summer term 2008 17 Movement Visualization • Also annotation with icons possible • Expressing speed by indicating mode of transportation • Moving by: − Foot − Car − Aircraft Summer term 2008 18 9 Annotation System • Annotations also manually applied to events by analyst − Highlight events − Direct attention of viewer • Choose from palette of annotations: − Curves, arrows, callouts, outlines, enlargement • Annotations implemented as data objects referenced to actual event data − Rather than separate graphical overlay − Semantic connection • Supports interactions such as mouseovers and menus on annotation graphics Summer term 2008 19 Annotation System • User annotation menu with five annotation styles Summer term 2008 20 10 Temporal Navigation • Events animated in time through 3D space as slider bar is moved • Selection of time range displayed − Years, weeks, days, hours, minutes • Allows perception of “who” and “what” in the “where” and “when” Summer term 2008 21 Stories in GeoTime • Stories system built on GeoTime time-space visualization framework • Introduces new sets of features: − Space-time pattern finding system − Search for common behaviors and relationships among events and entities − Capture management system − Stores and invokes snapshots of GeoTime space-time views • Provides specialized text editing panel (story window) − Allows analysts to author narratives and explanations − Links to bookmarked views − Essentially HTML text editor Summer term 2008 22 11 Story Window • Color-defined category • Categorize content to support collaboration or multiple story threads • Filter interface to hide or show text sections • Text sections titled ‘Pattern:…” generated by pattern functions • Click on link fetches and displays stored snapshot Summer term 2008 23 Stories in GeoTime • Supports representations for base elements of a story: − Events − People − Objects − Places − Relationships • Display elements in time and space -> Makes it suitable platform on which to build representations and interactions with stories Summer term 2008 24 12 Applications • Intelligence analysis − Criminal investigations − Analysis tradecraft training • Military • Law enforcement • Business • Historical visualizations Summer term 2008 25 Intelligence Data • GPS vehicle tracks • Personal transaction billing data • Phone calls • Credit card transactions • Proprietary operational data sets Summer term 2008 26 13 Story Visualization • Concept sketch of annotated GeoTime scene representing events of a fictional story Summer term 2008 27 The Little Red Riding Hood Summer term 2008 Picture: © Hoodwinked! 2005 28 14 The Little Red Riding Hood • Text of story "Little Red Riding Hood" • Meeting of Wolf and Red Riding Hood in forest • Visualize space-time paths (assuming GPS tracking data were available) • Point out: − Crossing paths by accident or − Wolf was waiting deliberately • Distinction immediately apparent in story visualization Summer term 2008 29 Behavior Patterns • Large volumes of low-level data − Extracting behaviors and patterns in order to understand situation − Common problem for analysts • Detection of patterns in movement activity − Discovery of possible meetings, speed of movement, percentage of time spent at unique locations, identifying gaps in observations Summer term 2008 30 15 Sense-Making • Evaluating alternative explanations accounting for some given evidence (important analysis technique) -> Competitive hypotheses Summer term 2008 31 Sense-Making Loop through Story Visualization Summer term 2008 32 16 Visual Analytics • Integration of scientific disciplines to improve division of labor between human and machine Summer term 2008 33 Story Spectrum • Relationship between story concepts and analytical workflow Summer term 2008 34 17 Evaluation • Assess new capabilities and usability • Formal evaluation performed by NIST (National Institute of Standards and Technology) showed: − High utility − High usability • Experienced intelligence analysts selected • Tested authoring and asynchronous communication of intelligence reports using GeoTime Stories − Integration of narrative text, snapshots, and pattern results into text report − Understandability by others (unfamiliar with situation and data) Summer term 2008 35 Evaluation Procedure • Participants presented with fictional scenario • Mimic scenarios encountered in a criminal investigation • Evaluation process divided into two stages: − Stage 1: Write report analyzing information in scenario − Stage 2: New scenario + review report authored by another analyst Summer term 2008 36 18 Example