Designing Multiple Relation Visualizations: Case Studies from Text Analytics Christopher Collins University of Ontario Institute of Technology
Duke University 5 April, 2013 Outline
2
Introduction
Research Method
Multiple Relation Visualization & Spatial Rights
Selected Projects
Looking to the Future Linguistic Visualization Divide
Card et al., 1999 Linguistic Visualization Divide
“the gulf separating sophisticated natural language processing algorithms and data structures from state-of-the-art interactive visualization design” “the gulf separating sophisticated natural language processing algorithms and data structures from state-of-the-art interactive visualization design” Bridging the Divide
Can linguistic data and algorithms, designed for NLP and CL purposes be re-appropriated to drive interactive visualizations of language? Successful InfoVis
7
Grounded in Reality Generalizable Research Method
8
Meet with data Evaluate experts discoveries •Basic grasp of data Design initial •Meet with data and questions visualization(s) experts for review
Acquire the data Revise •Analysis infrastructure visualization(s) •Meet with data experts for review Grounded evaluation is a process that attempts to ensure that the evaluations of information visualizations are situated within the context of intended use.
Grounded Evaluation for Information Visualizations Isenberg, P.; Zuk, T.; Collins, C. ; Carpendale, S. Proc. ACM BELIV 2008
9 design
evaluation implementation design
evaluation implementation Types of Evaluation
design
evaluation implementation
Pre-design evaluation of context informs Grounded evaluation of design and implementation Outline
13
Introduction
Research Method
Multiple Relation Visualization & Spatial Rights
Selected Projects
Looking to the Future Multi-Dimensional Data
14
Heer & boyd, 2005 Elmqvist et al., 2008 Multi-Dimensional Data
15
Heer & boyd, 2005 Elmqvist et al., 2008 Multi-Dimensional Data
16
Heer & boyd, 2005 Elmqvist et al., 2008 17 Spatial Rights
“The assignment of position as a visual encoding for a data dimension or relation.”
Collins and Carpendale, 2007 Spatial Encoding
18
Selective, Spatial Encoding
19
Selective,
Associative, Spatial Encoding
20
Selective, 7
Associative, 6 5 Ordered and Quantitative, 4 3
2
1
1 2 3 4 5 6 7 Spatial Encoding
21
Selective,
Associative,
Ordered and Quantitative,
Long in length , Spatial Encoding
22
Selective,
Associative,
Ordered and Quantitative,
Long in length ,
“Preattentive" Outline
23
Introduction
Research Method
Multiple Relation Visualization & Spatial Rights
Selected Projects
Looking to the Future Bubble Sets: Revealing Set Relations with Isocontours over Existing Visualizations 24 Christopher Collins, Gerald Penn, and Sheelagh Carpendale, InfoVis 2009
Simultaneous Spatial Rights
Primary Set Spatial Rights for Non-set Relation: Colouring
Primary Set Hybrid Spatial Rights: Rearrange
Primary Set Spatial Rights for Set Relation
Primary Set Spatial Rights for Non-set Relation: Convex Hull
Primary Set Spatial Rights for Non-set Relation: Bubble Set
Primary Set 34 Motivating Problem
Machine Translation Parse Trees Ethnographic Study
35
Contextual interviews with research team
Participatory observation
Artifact analysis
Goal is to describe meaning not make statistical inference
Thanks to our collaborators: Kevin Knight’s Statistical MT Research group @ ISI, University of Southern California Qualitative Analysis
36
Open Coding 6 interviews X 60-90 minutes transcribe code for interesting statements actively work against bias (don’t only seek answers to pre-determined questions) refine code set and review
(Strauss & Corbin, 1998) Complexity Brings Externalization
37
Very complex or very large: Cannot Complex: practically sketch sketch to aid manually understanding
Simple: analyze internally Visualization `in the Wild’
38
Some visualizations were in common use Large binders of parse trees and data tables Mostly in printed form Individual ad hoc design and usage Periodic analysis tasks mixed with programming Non-interactive Vocabulary differs from InfoVis community 39 Cooperative Design Sketches
40
Knight, 2007
Collins, 2008 Bubble Sets
41 Bubble Sets for Machine Translation Convex vs. Concave
42 Surface Routing
43 Order Effects
44
2 4 3 4
3 1 1 5 5 2 Order Effects
45
2 4 3 4
3 1 1 5 5 2 Order Effects
46
2 4 3 4
3 1 1 5 5 2
Calculating the Contour
48
? Calculating Contours Calculating Contours
Find Set Items Calculating Contours
Derive Virtual Edges Calculating Contours
Derive Active Region Calculating Contours
53
Calculate energy in active region Calculating Contours
54
Marching squares to find the contour Labelling Clusters
55
Labels along longest unobstructed virtual edge (before calculating energy) Heuristic Optimizations
56
Iterate until connected: 1-N: Reduce contour threshold: O(H+W) N-M: Increase positive weights and recalculate field: O(HWK)
For region of area H by W pixel groups containing K items.
58 Additional Applications 59 Hans Rosling, GapminderTrendalyzer 60 61 62 Hans Rosling, GapminderTrendalyzer 63 64 InfoVis Papers Over Time ([L]-Tat, 2007, [R]-Bubble Sets) 65 Challenges
66
Stability + speed → - stability (‘jiggle’ caused by super-pixel sizes) ‘hysteresis’ on virtual edges may reduce changes
Incremental updates recalculate surface only over subsets treat proximal items as a super-node
Obstacle Avoidance Failure Cannot guarantee non-set members are avoided When to give up trying? “by using the bubble sets representation [...] the researchers noticed the missing context information”
Smith, A.; Xu, W.; Sun, Y.; Faeder, J.R.; Marai, G.E. RuleBender: Integrated Visualization for Biochemical Rule-Based Modeling. IEEE Biovis Symposium 2011 [Best Paper Winner] -> University of Pittsburg
Strobelt, H.; Braun, J.; Deussen , O.; Groth, U.; Mayer, T.; Merhof, D. HiTSEE: A Visualization Tool for Hit Selection and Analysis in High-Throughput Screening Experiments. IEEE Biovis Symposium 2011 [Best Paper Runner Up] -> University of Konstanz
BioVis Symposium 2011 VisLink: Revealing Relationships Amongst Visualizations Christopher Collins and Sheelagh Carpendale, InfoVis 2007 Chang, M.-W.; Collins, C. 68 Exploring Entities in Text using Descriptive Non-Photorealistic Rendering. Pacific Vis, 2013. Brighton and Kirby, 2006
69 Linguistic Visualization Divide
(Collins, 2007) (Heer, 2006 [prefuse]) & (Fry, 2004) Understanding Multiple Relations
What is the relationship… across different views of the same data? across different relations in the same dataset? across multiple relations and datasets? VisLink Overview
Any number of 2D visualizations, each on its own plane in 3D space
Adjacent planes connected by bundled edges
Shortcuts and constrained widgets aid usability
Enables powerful inter-visualization queries Formalizing Multiple Relations Visualizations
Dataset Relation Visualization
Conference Attendee Data Professor / Student Node-link social network graph
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Dataset Relation Visualization
DA
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Dataset Relation Visualization
R D DA A ( A )
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Dataset Relation Visualization
R D Vis → R D DA A ( A ) A A ( A )
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Relation
RA (DA )
Dataset Visualization
Vis → R D DA A A ( A )
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Relation
RA (DA )
Dataset Visualization
Relation
D R D Vis → R D A B ( A ) A A ( A )
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Relation Visualization
VisA → RA (DA) RA (DA )
Dataset Visualization
Relation VisB → RA(DA ) D R D A B ( A )
Formalism for Multiple Relationship Visualization Comparison Formalizing Multiple Relations Visualizations
Relation Visualization
VisA → RA (DA) RA (DA )
Dataset Visualization
Relation VisB → RA(DA ) D R D A B ( A )
Visualization
Formalism for Multiple Relationship VisualizationVisC → RB (DA C)omparison Multiple Relation Visualizations
Individual Visualizations Coordinated Views Compound Graphs Semantic Substrates VisLink
Formalism for Multiple Relationship Visualization Comparison Individual Visualizations
Any datasets, relations, and visualizations
Manually compare
e.g. different charts in Excel
Formalism for Multiple Relationship Visualization Comparison Coordinated Views
Formalism for Multiple Relationship Visualization Comparison Coordinated Views
VisA → RA (DA )
Formalism for Multiple Relationship Visualization Comparison Coordinated Views
VisA → RA (DA ) VisA → RB (DA )
Formalism for Multiple Relationship Visualization Comparison Coordinated Views
VisA → RA (DA ) VisA → RB (DA )
Any datasets, relations, and visualizations Interactive highlighting e.g., Snap-Together Visualization (North & Shneiderman, 2000)
Formalism for Multiple Relationship Visualization Comparison Compound Graphs
Formalism for Multiple Relationship Visualization Comparison Compound Graphs
VisA → RA (DA )
Formalism for Multiple Relationship Visualization Comparison Compound Graphs
VisA → RA (DA ) + RB (DA )
Formalism for Multiple Relationship Visualization Comparison Compound Graphs
VisA → RA , RB (DA )
Secondary relation has no spatial rights e.g., Overlays on Treemaps (Fekete et al., 2003) , ArcTrees (Neumann et al., 2005), UseHierarchical of the powerful Edge spatialBundles dimension (Holten, 2006) to encode data relationships. Formalism for Multiple Relationship Visualization Comparison VisLink
Formalism for Multiple Relationship Visualization Comparison VisLink
VisA → RA (DA )
Formalism for Multiple Relationship Visualization Comparison VisLink
Vis → R D A A ( A ) VisB → RB (DA )
Formalism for Multiple Relationship Visualization Comparison VisLink
Vis → R D A A ( A ) VisB → RB (DA )
VisB → RA (DB )
Formalism for Multiple Relationship Visualization Comparison VisLink
Vis → R D A A ( A ) VisB → RB (DA )
VisA+B → T (RA (DA ), RB (DA ))
Formalism for Multiple Relationship Visualization Comparison VisLink
VisA+B → T (RA (DA ), RB (DA ))
Visualize second order relations between visualizations Across any datasets, relations, visualizations for which a relation can be defined All component visualizations retain spatial rights
Formalism for Multiple Relationship Visualization Comparison Equivalency & Extension
Formalism for Multiple Relationship Visualization Comparison Interaction With Component Visualizations
Always equivalent to 2D: Planes are virtual displays Mouse events transformed and passed to underlying visualization Equivalent to 2D viewing mode
VisLink Visualization Interplane Edges
VisLink Visualization Constrained Widget Interaction
VisLink Interaction 3D Navigation
VisLink Interaction Spreading Activation
Nodes have a level of activation , indicated by transparency of orange node background Full activation through: Selecting a node on a plane Node matches search query Activation propagates through interplane edges, reflecting between planes with exponential drop-off Enables inter-visualization queries Edge transparency relative to source node activation
Spreading Activation VisLink Case Study: Lexical Data
? WordNet IS-A hierarchy (R A) Similarity clustering (R B) using using radial tree (Vis A) force-directed layout (Vis B)
VisLink Visualization Inter-Plane Query Example
2: synonym sets 1: alphabetic clusters
No synonym information No alphabetic organization
Q: Synonyms in the alphabetic view?
Spreading Activation Inter-Plane Query Example
1. Select a word on plane 1
2. Edges propagate to synonym sets on plane 2
3. Reflected edges propagate back, revealing synonyms in alphabetic clusters
1: similarity clusters 2: synonym sets
Spreading Activation Inter-Plane Query Example
1. Select a word on plane 1
2. Edges propagate to synonym sets on plane 2
3. Reflected edges propagate back, revealing synonyms in alphabetic clusters
1: similarity clusters 2: synonym sets
Spreading Activation Inter-Plane Query Example
1. Select a word on plane 1
2. Edges propagate to synonym sets on plane 2
3. Reflected edges propagate back, revealing synonyms in alphabetic clusters
1: similarity clusters 2: synonym sets
Spreading Activation Inter-Plane Query Example
1. Select a word on plane 1
2. Edges propagate to synonym sets on plane 2
3. Reflected edges propagate back, revealing synonyms in alphabetic clusters
1: similarity clusters 2: synonym sets
Spreading Activation Edge Reflection and Inter-Plane Queries
Spreading Activation Linking Existing Visualizations
Case Study Linking Existing Visualizations
Case Study Perceptual Considerations
Not all layouts equal
Colour interactions with edges and visualizations
3D perspective bias
Conclusion Vlaming, L.; Collins, C. ; Hancock, M.; Nacenta, M.; Isenberg, T.; Carpendale, S. Integrating 2D Mouse Emulation with 3D Manipulation for Visualizations on a Multi-touch Table. Proceedings of ACM Interactive Tabletops and Surfaces, 2010. Masahiko Itoh, Naoki Yoshinaga, Masashi Toyoda, Masaru Kitsuregawa. Analysis and Visualization of Temporal Changes in Bloggers’ Activities and Interests. Proceedings of Pacific Vis, 2012. -> University of Tokyo
116 Barbara Fritsch and Wolfgang Knecht Graduate course project, 2011. -> TU Vienna
117 Interactive Exploration of Implicit and Explicit Relations
in Faceted Datasets Chang, M.-W.; Collins, C. 118 Jian Zhao, Christopher Collins,Exploring Fanny Entities Chevalier, in Text using an dDescriptive Ravin Balakrishnan, Non-Photorealistic 2013Rendering. Pacific Vis, 2013. Relations in Academic Papers
119
Explicit: Citation Reference
Implicit: Shared author Publication year Publication venue Keywords … Relations in Academic Papers
120
Explicit: Citation • Draw as links Reference
Implicit: Shared author • Imply through spatial layout Publication year Publication venue Keywords … Relations in Academic Papers
121
Explicit: Citation • Draw as links • Where possible, Reference layout to remove edge crossings Implicit: Shared author • Imply through spatial layout Publication year Publication venue Keywords … 122 123 124
125 Outline
126
Introduction
Research Method
Multiple Relation Visualization & Spatial Rights
Selected Projects
Looking to the Future Sketch-based Interfaces
127 NUI Interaction Challenges
128
Occlusion issues on touch interfaces
Orientation (tables) and distance (walls)
Collaboration considerations
Text entry
Direct Manipulation Mixed Data
Audio and Text
Illustrated Books
SMS and Location
Tweets and Time GRAD RAFAEL VERAS DANIEL CHANG ERIK PALUKA BRITTANY KONDO
UNDERGRAD BENJAMIN WATERS TANVIR KAYKOBAD GOWTH MAHESWARA BRADLEY CHICOINE ZACHARY COOK TONY TRAN
SHEELAGH CARPENDALE GERALD PENN MARTIN WATTENBERG FERNANDA VIÉGAS UTA HINRICHS PETRA ISENBERG MARK HANCOCK FANNY CHEVALIER JEFFREY HEER JIAN ZHAO JULIE THORPE JEREMY BRADBURY DAN MCFARLAND MARIAN DÖRK
130