Chapter 3 VISUAL REASONING AND REPRESENTATION Boris Kovalerchuk Central Washington University, USA Abstract: Reasoning plays a critical role in decision making and problem solving. This chapter provides a comparative analysis of visual and verbal (sentential) rea- soning approaches and their combination called heterogeneous reasoning. It is augmented with a description of application domains of visual reasoning. Specifics of iconic, diagrammatic, heterogeneous, graph-based, and geometric reasoning approaches are described. Next, explanatory (abductive) and deduc- tive reasoning are identified and their relations with visual reasoning are ex- plored. The rest of the chapter presents a summary of human and model-based reasoning with images and text. Issues considered include: cognitive opera- tions, difference between human visual and spatial reasoning, and image rep- resentation. One of the main our statements in this chapter is that the funda- mental iconic reasoning approach proclaimed by Charles Peirce is the most comprehensive heterogeneous reasoning approach. Key words: Visual reasoning, spatial reasoning, heterogeneous reasoning, iconic reason- ing, explanatory reasoning, geometric reasoning. The words or the language, as they are written or spoken, do not seem to play any role in my mechanism of thought. Albert Einstein 1. VISUAL VS. VERBAL REASONING Scientists such as Bohr, Boltzmann, Einstein, Faraday, Feynman, Hei- senberg, Helmholtz, Herschel, Kekule, Maxwell, Poincare, Tesla, Watson, and Watt have declared the fundamental role that images played in their most creative thinking [Thagard & Cameron, 1997; Hadamard, 1954; Shep- ard & Cooper, 1982]. 50 Chapter 3 Problem solving and decision making is based on reasoning, where the result of such reasoning is a solution or decision. Herbert Simon [Simon, 1995] pointed out that Aristotelian logic and Euclidean geometry were major and abiding contributions of the Greeks to reasoning in language (natural or formal) and drawing inferences from diagrams and other pictorial sources to solve problems of logic and geometry. Note that despite the frequent refer- ences to Greek mathematics as an origin of visual reasoning, the Chinese and Indians knew a visual proof of the Pythagorean Theorem in 600 B.C. before it was known to the Greeks. This visual proof is shown in Figure 1 [Kulpa, 1994]. Figure 1. Pythagorean Theorem known to the Chinese and Indians [Kulpa, 1994] It is widely acknowledged that visual diagrammatic, iconic representa- tions of reasoning are better understood than verbal explanations because diagrams and icons are symbols that better resemble what they represent than text [Thagard & Cameron, 1997]. Simon [Simon, 1995] discusses the heuristic nature of human reasoning in problem solving as an argument for using non-verbal visual reasoning using diagrams as a tool to foster reasoning and to find answers. According to Simon, traditional reasoning in non-visual formal logic is much more helpful in testing the correctness of the reasoning than in identifying the statement to be inferred (finding a solution). Another direction to consider for a visual approach is to foster problem finding in the space of alternative problems vs. problem solving that we dis- cussed above. Einstein and Infeld [Einstein & Infeld, 1938] stated: The formulation of a problem is often more essential than its solution, which may be merely a matter of mathematical or experimental skill. To raise new questions, new possibilities, to regard old problems from a new angle, requires creative imagination and marks real advance in science. It is well known this process is extremely informal not only in scientific discovery but in design (architectural and others). Several attempts have 3. Visual Reasoning and Representation 51 been made to provide a comprehensive structural picture of this informal visual process. Hepting [Hepting, 1999] formulated this picture as a collec- tion of 16 design principles. We structured them into two processing catego- ries: I. Support for a model of problem solving by: (1) supporting restruc- turing and reorganizing of alternatives, (2) encouraging the user to discover things personally, (3) supporting the systematic exploration of a conceptual space, (4) allowing the user to concentrate on the task, (5) providing an external representation of the possible choices, and (6) allowing each user to apply personal judgment. II. Technical tools to support problem solving by: (1) supporting the user in working directly with images, (2) supporting the user in com- bining images, and their elements, (3) allowing the user to employ heuristic search techniques, (4) recording all aspects of the visual representations of the design, (5) supporting collaboration, (6) assist- ing the user in choosing images, (7) allowing the user to interact with the images, (8) supporting multiple visual representations, and (9) supporting the use of current solutions in future enquiries. Once again, this list shows that the process is very informal (and proba- bly sometimes illogical). It is also illustrates the huge role of visual reason- ing in all stages of the design process. Another argument for visual reasoning is that more than one medium provides information for reasoning that includes text and pictures, but formal logic is limited by sentences [Shin & Lemon, 2003]. This argument is used for supporting both pure visual reasoning and for heterogeneous reasoning that combines text, pictures, and potentially any other medium [Barwise & Etchemendy, 1995]. The next argument for visual and heterogeneous reasoning is related to the speed and complexity of reasoning. Reasoning with diagrams and with- out re-expressing the diagrams in the form of a sentence can be simpler and faster. It avoids an unnecessary and non-trivial information conversion process, by working directly with heterogeneous rules of inference, e.g., First Order Logic and Euler/Venn reasoning [Swoboda & Allwein, 2002; Swoboda & Barwise, 2002]. 2. ICONIC REASONING One of the founders of the modern formal logic Charles Peirce (1839- 1914) argued for use of visual inference structures and processes a long time 52 Chapter 3 ago. Recently it has become increasingly clear that one of the fundamental difficulties of automatic computer reasoning is that it is extremely hard to incorporate the human observational and iconic part of the reasoning pro- cess into the computer programs. Indeed, the extreme opinion is that it is simply impossible [Tiercelin, 1995]. Peirce stated that in order for symbols to convey any information, indices and icons must accompany them [Peirce, 1976; Hartshorne, Weiss & Burks, 1958; Robin, 1967; Tiercelin, 1995]. It is probably not accidental that in ad- dition to being a logician and a mathematician, Peirce was also a land sur- veyor at the American Geodesic Coast Survey. In this capacity, he would have first hand experience with real world visual and spatial geographic rea- soning. We should note here that several chapters in this book deal with vis- ual and spatial reasoning and problem solving related to combining geo- graphic maps, aerial and satellite photos. Charles Peirce distinguished the iconic, indexical and symbolic functions of signs. Table 1, based on [Tier- celin, 1995], summarizes Peirce’s view of icons and diagrams. Table 1. Peirce’s concepts of icons and diagrams Icons Icon: an object that may be purely fiction, but must be logically possible Main icon function: exemplification or exhibition of an object (its characteris- tics) Secondary icon function: resemblance to the object By direct observation of an icon, other truths concerning its object can be dis- covered. The ideal iconic experimentation warrants an accord between the model and the original. Icons are formal and not merely empirical images of the things. Icons represent the formal side of things. Diagrams species of icons According to Peirce, in general, mathematical reasoning and deduction involve appeal to the observation of “iconic” representations and it cannot be reduced to purely symbolic (i.e. rule-governed) transformations [Peirce, 1933; Ransdell, 1998]. In modern studies, the term “diagrammatic” princi- pally replaced the Peirce’s term “iconic”. Peirce stated that all thinking is in signs, and that signs can be icons, indices, or symbols [Thagard & Cameron, 1997; Goudge, 1950; Hartshorne & Weiss, 1958]. We believe that Peirce’s original term “iconic” has an important flavor that has faded in the modern, more “scientific” term “diagrammatic”. We discuss iconic representations further in Chapters 9 and 10. Diagrammatic reasoning is an active area of research with a variety of subjects being considered [Shin & Lemon, 2003; Hegarty, Meyer & Nara- yanan, 2002; Chandrasekaran, Josephson, Banerjee, Kurup & Winkler, 2002; Chandrasekaran, 2002; Chandrasekaran, 1997; Anderson, Meyer & Ovier, 2002; Anderson, 1999; Magnani, Nersessian & Pizzi, 2002; Glasgow, Chan- 3. Visual Reasoning and Representation 53 drasekaran & Narayanan, 1995; Chandrasekaran & Narayanan, 1990; Bar- wise & Etchemendy, 1999; Barwise & Etchemendy, 1998; Barwise & Etchemendy, 1995; Swoboda & Allwein, 2002; D’Hanis, 2002; Shimojima, 2002; Magnani, 1999]. Applications of diagrammatic reasoning range from teaching introducto- ry logic classes [Barwise & Etchemendy, 1994] to motions of vehicles and individuals engaged in a military exercise [Chandrasekaran, et al., 2002]. There is no full consensus on the definition of a diagram. Shin and Lem- on [Shin & Lemon, 2003]