A Formalism for Visual Query Interface Design by Jiwen Huo A thesis presented to the University of Waterloo in fulfillment of the thesis requirement for the degree of Doctor of Philosophy in Computer Science Waterloo, Ontario, Canada, 2008 c Jiwen Huo 2008 Library and Archives Bibliothèque et Canada Archives Canada Published Heritage Direction du Branch Patrimoine de l’édition 395 Wellington Street 395, rue Wellington Ottawa ON K1A 0N4 Ottawa ON K1A 0N4 Canada Canada Your file Votre référence ISBN: 978-0-494-55514-9 Our file Notre référence ISBN: 978-0-494-55514-9 NOTICE: AVIS: The author has granted a non- L’auteur a accordé une licence non exclusive exclusive license allowing Library and permettant à la Bibliothèque et Archives Archives Canada to reproduce, Canada de reproduire, publier, archiver, publish, archive, preserve, conserve, sauvegarder, conserver, transmettre au public communicate to the public by par télécommunication ou par l’Internet, prêter, telecommunication or on the Internet, distribuer et vendre des thèses partout dans le loan, distribute and sell theses monde, à des fins commerciales ou autres, sur worldwide, for commercial or non- support microforme, papier, électronique et/ou commercial purposes, in microform, autres formats. paper, electronic and/or any other formats. The author retains copyright L’auteur conserve la propriété du droit d’auteur ownership and moral rights in this et des droits moraux qui protège cette thèse. Ni thesis. Neither the thesis nor la thèse ni des extraits substantiels de celle-ci substantial extracts from it may be ne doivent être imprimés ou autrement printed or otherwise reproduced reproduits sans son autorisation. without the author’s permission. In compliance with the Canadian Conformément à la loi canadienne sur la Privacy Act some supporting forms protection de la vie privée, quelques may have been removed from this formulaires secondaires ont été enlevés de thesis. cette thèse. While these forms may be included Bien que ces formulaires aient inclus dans in the document page count, their la pagination, il n’y aura aucun contenu removal does not represent any loss manquant. of content from the thesis. I hereby declare that I am the sole author of this thesis. This is a true copy of the thesis, including any required final revisions, as accepted by my examiners. I understand that my thesis may be made electronically available to the public. ii Abstract The massive volumes and the huge variety of large knowledge bases make infor- mation exploration and analysis difficult. An important activity is data filtering and selection, in which both querying and visualization play important roles. Interfaces for data exploration environments normally include both, integrating them as tightly as possible. But many features of information exploration environments, such as visual rep- resentation of queries, visualization of query results, interactive data selection from visualizations, have only been studied separately. The intrinsic connections between them have not been described formally. The lack of formal descriptions inhibits the development of techniques that produce new representations for queries, and natural integration of visual query specification with query result visualization. This thesis describes a formalism that describes the basic components of informa- tion exploration and and their relationships in information exploration environments. The key aspect of the formalism is that it unifies querying and visualization within a single framework, which provides a foundation for designing and analysing visual query interfaces. Various innovative designs of visual query representations can be derived from the formalism. Simply comparing them with existing ones is not enough, it is more im- portant to discover why one visual representation is better or worse than another. To do this it is necessary to understand users’ cognitive activities, and to know how these cognitive activities are enhanced or inhibited by different presentations of a query so that novel interfaces can be created and improved based on user testing. This thesis presents a new experimental methodology for evaluating query repre- sentations, which uses stimulus onset asynchrony to separate different aspects of query comprehension. This methodology was used to evaluate a new visual query represen- tation based on Karnaugh maps, and showing that there are two qualitatively different approaches to comprehension: deductive and inductive. The Karnaugh map represen- tation scales extremely well with query complexity, and the experiment shows that its good scaling properties occur because it strongly facilitates inductive comprehension. iii Acknowledgements There are many people whose support, guidance, and encouragement have made this thesis possible. I am especially grateful to my advisor, William Cowan, for his advice, support, and guidance throughout my study in Waterloo. He has taught me so much and been a great mentor. In addition, I would like to thank my reading committee, Christopher Healey, Catherine Burns, Grant Weddell, and Ric Holt, for the time and effort they spent to read and comment on my work. Also, I would like to thank all of people who participated in my experiments for their time and attention. Special acknowledgement goes to Yi Lin, Jie Xu, and Yinbin Liu, for giving me feedbacks and suggestions on my pilot study to improve the experiment. Most of all, I would like to thank my parents and my husband Chong for always supporting and encouraging me. Without their love and support, the completion of this thesis would not be possible. Finally I’d like to thank my son Ray for the joy he brings to my life. iv Contents List of Tables 1 List of Figures 1 1 Introduction 1 1.1 Contributions . 2 1.2 User Interface Issues in Information Exploration . 3 1.3 A Formalism for Visual Query Interface . 4 1.4 Issues in Evaluating Query Languages . 5 1.5 The Experiment Methodology . 6 1.6 Thesis Organization . 7 2 Related Work on Visual Information Exploration 9 2.1 Information Exploration . 9 2.1.1 Information Exploration Process . 9 2.1.2 Visual Information Exploration . 11 2.2 Information Visualization . 13 2.2.1 The Process of Visualization . 13 2.2.2 Data Model . 14 2.2.3 Visual Mapping . 15 2.2.4 Data Selection and Visualization . 16 2.3 Boolean Query Specification . 17 2.3.1 Boolean Expressions and Truth-Tables . 18 2.3.2 Command Languages and Natural Language for Data Querying 19 2.3.3 Form-Based Query and Dynamic Query Technique . 21 v 2.3.4 Graphical Approaches to Query Specification Based on Boolean Expressions . 21 2.3.5 Graphical Approaches to Query Specification Based on Truth- Table................................. 23 2.4 Query Result Display . 25 2.4.1 Query Preview and Immediate Updates of Query Results . 26 2.4.2 Context Information of Data Distribution along Each Attribute 27 2.4.3 Rank of Query Results . 27 2.4.4 The Relationship of Query Results with Query Terms . 27 2.5 Interactive Data Selection on Visual View . 28 2.6 Formal Models of Visual Information Exploration . 30 3 A Formalism for Visual Query Interfaces 33 3.1 The Table Model . 33 3.1.1 Definition of the Table Model . 34 3.1.2 Applying Tables to Tables . 35 3.1.3 Data Visualization by Tables . 36 3.1.4 Table Model for Queries . 37 3.1.5 Visualizing Queries . 38 3.2 The Transformation Model . 40 3.2.1 Query on Data Space . 41 3.2.2 Visual Query on Visual Representations . 42 3.3 Query Result Visualization . 44 3.3.1 Traditional Method of Query Result Visualization . 44 3.3.2 Example: Enhanced Visualization of Query Result . 45 3.3.3 The Transformation Model of Query Result Visualization . 45 3.3.4 Examples of Query Result Visualization . 47 3.4 Visualizing the Mappings . 48 3.4.1 Visualizing the Visualization Mappings . 48 3.4.2 Visualizing the Query Mappings . 50 3.5 The Recursive Models . 52 3.5.1 Visualizing Visual Representations . 53 3.5.2 Querying on Queries . 54 3.5.3 The Iterative Model . 54 vi 3.6 Summary . 56 4 The Application System KMVQL 58 4.1 Software Design based on the Formalism . 58 4.1.1 User Actions . 59 4.1.2 Software Domain Model . 60 4.1.3 Module Responsibilities . 61 4.1.4 Summary of the Software Model . 63 4.2 Overview of KMVQL . 64 4.2.1 Visualization of Query and Query Results . 64 4.2.2 Visualization of Data Space . 66 4.2.3 Query Device . 67 4.2.4 Information Seeking Nodes . 68 4.2.5 Formulate Boolean Queries in KMVQL . 68 4.2.6 Compound Query based on Intermediate Results . 69 4.3 Example of Exploring Data with KMVQL . 71 4.4 Summary . 75 5 Related Work on Query Language Evaluation 76 5.1 Introduction to Query Language Evaluation . 76 5.1.1 Framework of Variables for Query Language Evaluation . 77 5.1.2 Tasks for Query Language Evaluation . 79 5.1.3 Boolean Concept Complexity . 79 5.2 Review of Related Experiments . 82 5.2.1 Example Analysis: TEBI . 82 5.2.2 Example Analysis: Filter/Flow . 84 5.2.3 Example Analysis: Chui’s User Study on Query Languages . 84 5.2.4 Summary of Previous Experiments . 86 6 A Methodology for Evaluating Visual Query Languages 89 6.1 Introduction . 89 6.2 Method . 90 6.2.1 Query Task . 90 6.2.2 Components of Query Task . 91 vii 6.3 Isolating the Time Components . 91 6.3.1 Manipulating Stimulus Onset Asynchrony . 92 6.3.2 A Possible Experimental Hypothesis . 92 6.3.3 Data Fitting . 94 7 The Experiment 96 7.1 Introduction . 96 7.2 Method . 97 7.2.1 Query Task . 97 7.2.2 Experiment Trials . 98 7.2.3 Queries .