1

Symbol Grounding Problem Angelo Loula1,2 & João Queiroz 2,3* 1 Informatics Area, Exact Sciences Department, State University of Feira de Santana, Brazil. 2 Department of Computer Engineering and Industrial Automation, School of Electrical and Computer Engineering, State University of Campinas (UNICAMP), Brazil. 3 Research Group on History, Philosophy, and Biological Sciences Teaching, Institute of Biology, Federal University of Bahia, Brazil. * Corresponding Author [email protected]

1 THIS PREPRINT WAS PUBLISHED AS: Loula, A., Queiroz, J. (2008) Symbol grounding problem. In: Rabunal, J., Dorado, J., Pazos, A. (eds.) Encyclopedia of Artificial Intelligence, pp. 1543–1548. IGI Global

INTRODUCTION The topic of representation acquisition, manipulation and use has been a major trend in Artificial Intelligence since its beginning and persists as an important matter in current research. Particularly, due to initial focus on development of symbolic systems, this topic is usually related to research in symbol grounding by artificial intelligent systems. Symbolic systems, as proposed by Newell & Simon (1976), are characterized as a high-level cognition system in which symbols are seen as “[lying] at the root of intelligent action” (Newell and Simon, 1976, p.83). Moreover, they stated the Physical Symbol Systems Hypothesis (PSSH), making the strong claim that “a has the necessary and sufficient means for general intelligent action” (p.87).

This hypothesis, therefore, sets equivalence between symbol systems and intelligent action, in such a way that every intelligent action would be originated in a symbol system and every symbol system is capable of intelligent action. The symbol system described by Newell and Simon (1976) is seen as a computer program capable of manipulating entities called symbols, ‘physical patterns’ combined in expressions, which can be created, modified or destroyed by syntactic processes. Two main capabilities of symbol systems were said to provide the system the properties of closure and completeness, and so the system itself could be built upon symbols alone (Newell & Simon, 1976). These capabilities were designation – expressions designate objects – and interpretation – expressions could be processed by the system. The question was, 2 and much of the criticism about symbol systems came from it, how these systems, built upon and manipulating just symbols, could designate something outside its domain.

Symbol systems lack ‘’, stated (1980), in an important essay in which he described a widely known mental experiment (Gedankenexperiment), the ‘ Argument’. In this experiment, Searle places himself in a room where he is given correlation rules that permits him to determine answers in Chinese to question also in Chinese given to him, although Searle as the interpreter knows no Chinese. To an outside observer (who understands Chinese), the man in this room understands Chinese quite well, even though he is actually manipulating non-interpreted symbols using formal rules. For an outside observer the symbols in the questions and answers do represent something, but for the man in the room the symbols lack intentionality. The man in the room acts like a symbol system, which relies only in symbolic structures manipulation by formal rules. For such systems, the manipulated tokens are not about anything, and so they cannot even be regarded as representations. The only intentionality that can be attributed to these symbols belongs to who ever uses the system, sending inputs that represent something to them and interpreting the output that comes out of the system. (Searle, 1980)

Therefore, intentionality is the important feature missing in symbol systems. The concept of intentionality is of aboutness, a “feature of certain mental states by which they are directed at or about objects and states of affairs in the world” (Searle, 1980), as a thought being about a certain place. 1 Searle (1980) points out that a ‘program’ itself can not achieve intentionality, because programs involve formal relations and intentionality depends on causal relations. Along these lines, Searle leaves a possibility to overcome the limitations of mere programs: ‘machines’ – physical systems causally connected to the world and having ‘causal internal powers’ – could reproduce the necessary causality, an approach in the same direction of situated and embodied and robotics. It is important to notice that these ‘machines’ should not be just robots controlled by a symbol system as described before. If the input does not come from a keyboard and output goes to a monitor, but rather came in from a video camera and then out to motors, it would not make a difference since the symbol system is not aware of this change. And still in this case, the robot would not have intentional states (Searle 1980).

1 See also Dennett & Haugeland 1987, Searle 1983, Jacob 2003. 3

Symbol systems should not depend on formal rules only, if symbols are to represent something to the system. This issue brought in another question, how symbols could be connected to what they represent, or, as stated by Harnad (1990) defining the Symbol Grounding Problem:

“How can the semantic interpretation of a formal symbol system be made intrinsic to the system, rather than just parasitic on the meanings in our heads? How can the meanings of the meaningless symbol tokens, manipulated solely on the basis of their (arbitrary) shapes, be grounded in anything but other meaningless symbols?”

The Symbol Grounding Problem, therefore, reinforces two important matters. First that symbols do not represent anything to a system, at least not what they were said to ‘designate’. Only someone operating the system could recognize those symbols as referring to entities outside the system. Second, the symbol system cannot hold its closure in relating symbols only with other symbols; something else should be necessary to establish a connection between symbols and what they represent. An analogy made by Harnad (1990) is with someone who knows no Chinese but tries to learn Chinese from a Chinese/Chinese dictionary. Since terms are defined by using other terms and none of them is known before, the person is kept in a ‘dictionary-go-round’ without ever understanding those symbols.

The great challenge for Artificial Intelligence researchers then is to connect symbols to what they represent, and also to identify the consequences that the implementation of such connection would make to a symbol system, e.g. much of the descriptions of symbols by means of other symbols would be unnecessary when descriptions through grounding are available. It is important to notice that the grounding process is not just about giving sensors to an artificial system so it would be able to ‘see’ the world, since it ‘trivializes’ the symbol grounding problem and ignores the important issue about how the connection between symbols and objects are established (Harnad, 1990).

BACKGROUND The symbol grounding problem aroused from the notice that symbol systems manipulated structures that could be associated with things in the world by an observer operating the system, but not by the system itself. The quest for symbol grounding processes is concerned with 4 understanding processes which could enable the connection of these purely symbolic representations with what they represent in fact, which could be directly, or by means of other grounded representations. This represents a technological challenge as much as a philosophical and scientific one, but there is a strong interrelation between them. From one side there is the concern with the technological design and engineering of symbol grounding processes in artificial systems. On the other side, the grounding process is a process present in natural systems and therefore precedes artificial systems. Theories and models are developed to explain grounding and if consistent and detailed enough may in principle be implemented in artificial systems, which in return correspond to a laboratory for these theories, when their hypothesis are tested and new questions are raised, allowing further refinement and experimentation. A first proposal for symbol grounding as made by Harnad (1990) in the same paper where he gave a definition for the ‘symbol grounding process’. Harnad proposed that symbolic representations should be grounded bottom-up by means of non-symbolic representations: iconic representations – sensory projections of objects – and categorical representations – invariant features of objects. Neural networks were pointed out as a feature learner and discriminator, which could link sensory data with symbolic representations, after been trained to identify the invariant features. This would causally connect symbols and sensory data, but this proposal describes just a tagging system that gives names to sensed objects but does not use this to take actions and interact with its environment. A ‘mental theater’ is formed as Dennett (1991) defined, where images are projected internally and associated with symbols, but no one is watching it. Besides the symbols and the iconic representations are probably given by the systems operator and the system must learn them all, making no distinction between them and attributing no functionality to them. Another approach to deal with the limitation of symbol systems was presented by Brooks (1990). Instead of modeling artificial systems as symbol systems, Brooks rejected the symbolic approach for cognition modeling and the need of representations for this end: “[r]epresentation is the wrong unit of abstraction in building the bulkiest parts of intelligent systems”(Brooks, 1991b, p.139), with representations seen as centralized, explicit and pre-defined structures. He proposed the Physical Grounding Hypothesis (Brooks, 1990), supporting that intelligent systems should be embedded in the real world, sensing and acting in it, establishing causal relations with 5 perceptions and actions, being built in a bottom up manner with higher levels depending on lower ones. There was no need for representations because the system is already in touch with the objects and events it would need to represent. Moreover, Brooks called up attention that the most important aspect of intelligence was left out: to deal with the world and its dynamics. Instead of dealing with sophisticated high-level processes dealing with simple domains, research in the so- called Nouvelle AI should focus in simpler processes dealing with greatly complicated domains (such as the real world) and work its way from there to higher level ones (Brooks, 1990). Brooks (1991a) also stated principles to this new approach, such as situatedness and embodiment, which are the mottos of the situated and embodied cognition studies (Clark, 1997). Symbolic representations are in fact not incompatible with the Physical Grounding Hypothesis and with the Situated and Embodied Cognition approach. Brooks (1990) himself pointed out high-level abstractions should be made ‘concrete’ by means lower-level processes, thus symbolic representations should be causally constructed from the situated/embodied interaction dynamics through the artificial agent’s history. This approach was followed by several researchers dealing with symbol grounding when building artificial systems in which representations emerge from agent’s interactions, when learning processes take place (e.g. Ziemke 1999, Vogt 2002, Cangelosi 2002, Roy 2005; see also Christiansen & Kirby 2003; Wagner, 2003, for a review of experiments about language emergence). In most of these new systems, artificial agents are situated in an environment, in which they can sense and act, and are allowed to interact with other agents – either artificial or biological ones. By means of associative learning mechanisms, agents are able to gradually establish relations between representations and what they represent in the world, using communication as the basis to guide this learning process. And remarkably, when explicitly discussing the ‘symbol grounding problem’ (Vogt 2002, Cangelosi et al. 2002, Roy 2005), the theory of , particularly his definition of a symbol, is brought forth as the theoretical background for a new view into this problem.

SEMIOTICS AND SYMBOL GROUNDING PROBLEM The symbol grounding problem is fundamentally a matter of how certain things can represent other things to someone. Although symbol systems were said to have ‘designation’ properties (Newell & Simon, 1976), which would allow symbols manipulated by the system to 6 stand for objects and events in the world, this property should actually be attributed to an outside observer who was the only one able to make this connection. The artificial system itself did not have this capability, so the symbols it manipulated were said to be ungrounded. Building artificial systems based on the hypothesis that symbolic processes were autonomous and no other process was required proved to be flawed, and the quest to understand representation processes came up as a major issue. Representation is the focus of , the ‘formal science of ’ as defined by Charles Sanders Peirce. His definition of Semiotics and his pragmatic notion of meaning as the ‘action of signs’ (semiosis), have had deep impact in philosophy, psychology, theoretical biology, and cognitive sciences. Sign model and classification was developed by Peirce from his logical- phenomenological categories. His definition of a sign as “something which stands to somebody for something in some respect or capacity” (Peirce 1931-1958, §2.228) interrelates three distinct elements: a sign, an object, which the sign represents in some respect, and an effect (interpretant) on an interpreter. The nature of the relation between sign and object establishes the ‘most fundamental division of signs’: signs can either be icons, indexes or symbols. Icons stand for the object through resemblance or similarity, since it carries properties in common with the object. The drawing of an object, a diagram and ‘sensory projections’ are regarded as icons. Indexes establish spatio-temporal physical relation with its object, both occur as events and the interpreter is not responsible for connection between them, he just remarks it when it is established (Peirce 1931-1958, §2.299). Examples of indexes are smoke, which is related with fire, a scream that calls up our attention, or a bullet hole. According to Peirce, a symbol is a sign because it is interpreted as such, due to a natural or conventional disposition, in spite of the origin of this general interpretation rule (Peirce 1931-1958, §2.307). A word, a text and even a red light in a traffic light alerting drivers to stop are symbols. In this symbolic process, the object which is communicated to the interpretant through the sign is a lawful relationship between a given type of sign and a given type of object. Generally speaking, a symbol communicates a law to the interpretant as a result of a regularity in the relationship between sign and object. Furthermore, it is important to remark that symbols, indexes and icons are not mutually exclusively classes; they are interrelated and interdependent classes. “A Symbol is a law, or regularity of the indefinite future. […] But a law necessarily governs, or “is embodied in” individuals, and prescribes some of their qualities. Consequently, a constituent of a Symbol may 7 be an Index, and a constituent may be an Icon” (Peirce 1931-1958, §2. 293). Symbols require indexes which require icons. Harnad (1990) already noticed that symbols need non-symbolic representations and proposed that symbols are to be connected with sensory projections and categorical features, both regarded as icons in Peirce’s theory. A symbol is a sign and as such it involves an object which it refers to and an interpretant, the effect of the sign, so a symbol can only represent something to someone and when someone is interpreting it. A symbol distinguishes itself from other signs since it holds no resemblance or spatial-temporal relation with the object and thus depends on a general rule or disposition from the interpreter. At last, symbols incorporate indexes (and icons, consequently) and one way symbols can be acquired is by exploiting indexical relations between signs and objects, establishing regularities between them.

FUTURE TRENDS The discussion around the symbol grounding problem has an important component of theoretical aspects since it involves issues such as representation and cognitive modeling. Nevertheless, it is also a technological concern if researchers in Artificial Intelligence intend to model and build artificial systems which are capable of handling symbols in the appropriate way. The most evident consequence of discussion the problem of symbol grounding and ways of solving it is related to language and more generally with communication systems between agents (artificial ones or not). If we expect a robot to act appropriately when we say ‘bring me that cup’, we should expect it to know what these symbols represent so it will act accordingly. Moreover, we expect an artificial agent to learn and establish symbol-object connections autonomously, without the need of programming everything prior to the robot execution or reprogram it every time a new symbol is to be learned. The employment of strong theoretical basis that describes thoroughly the process of interest can certainly contribute to the endeavor of modeling and implementing it in artificial systems. The semiotics of Charles S. Peirce is recognized as a strongly consistent theory, and has been brought forth by diverse researchers in Artificial Intelligence, though fragmentally. We expect that Peirce description of sign processes will shed light on the intricate problem of symbol grounding. Particularly, Peirce conception of meaning as sign action can open up perspectives on the implementation of semiotic machines, which can produces, transmits, receives, computes, and 8 interprets signs of different kinds, meaningfully (Fetzer 1990). According to the pragmatic approach of Peirce, meaning is not an infused concept, but a power to engender interpretants (effects on interpreters). According to Peirce’s pragmatic model of sign, meaning is a, context- sensitive (situated), interpreter-dependent, materially extended (embodied) dynamic process. It is a social-cognitive process, not merely a static system. It emphasizes process and cannot be dissociated from the notion of a situated (and actively distributed) communicational agent. It is context-sensitive in the sense that is determined by the network of communicative events within which the interpreting agents are immersed with the signs, such that they cooperate with one another. It is both interpreter-dependent and objective because it triadically connects sign, object, and an effect in the interpreter.

CONCLUSION The original conception of artificial intelligent systems as symbols systems brought forth a problem known as symbol grounding problem. If symbol systems manipulate symbols, these symbols should represent something to the system itself and not only to an external observer, but the system has no way of grounding these symbols in its sensory and motor interaction history, since it does not have one. Many researchers pointed this key flaw, particularly John Searle (1980) with his Chinese Room Argument and Stevan Harnad (1990) with the definition for the problem became well know. A direction towards modeling artificial systems as embodied and situated agents instead of symbol manipulating systems was pointed out, urging the need to implement systems that could autonomously interact with its environment and with the things it should have representations for. But the topic of symbol grounding also needs a description of how certain things come to represent other things to someone, the topic of study of semiotics. The semiotics of C.S.Peirce has been used as theoretical framework in the discussion the topic of symbol grounding problem in Artificial Intelligence. The application of his theory in dealing with the symbol grounding problem should further contribute to the development of computational models of cognitive systems and to the construction of ever more meaningful machines.

REFERENCES Brooks, R. A. (1990) Elephants Don't Play Chess. Robotics and Autonomous Systems (6), 3-15. 9

Brooks, R. A. (1991a) Intelligence Without Reason. Proceedings of 12th Int. Joint Conf. on Artificial Intelligence, Sydney, Australia, August 1991, 569-595.

Brooks, R. A. (1991b) Intelligence Without Representation. Artificial Intelligence Journal (47), 139–159.

Cangelosi, A., Greco, A. & Harnad, S. (2002). Symbol grounding and the symbolic theft hypothesis. In A. Cangelosi & D. Parisi (Eds.), Simulating the Evolution of Language (chap.9). London:Sprinter.

Christiansen, M.H. & Kirby, S. (2003). Language evolution: consensus and controversies. Trends in Cognitive Sciences, 7 (7), 300-307.

Clark, A. (1997) Being There: Putting Brain, Body and World Together Again. Cambridge MA: MIT Press, 1997.

Dennett, D. (1991), Explained. Boston : Little, Brown and Co.

Dennet, D., Haugeland, J. (1987) Intentionality. In R. L. Gregory (Ed.), The Oxford Companion to the Mind, Oxford University Press.

Harnad, S. (1990) The Symbol Grounding Problem. Physica D 42: 335-346.

Haugeland, J. (1985) Artificial Intelligence: the Very Idea. The MIT Press: Cambridge, Massachusetts.

Newell, A., & Simon, H.A. (1976). Computer science as empirical inquiry: Symbols and search. Communications of the Association for Computing Machinery, 19(3), 113-126.

Peirce, C.S. (1931-1958) The Collected Papers of Charles Sanders Peirce. Vols. 1-6, Charles Hartshorne and Paul Weiss (Eds.), Vols. 7-8, Arthur W. Burks (Ed.). Cambridge, Mass.: Harvard University Press.

Roy, D. (2005) Semiotic Schemas: A Framework for Grounding Language in Action and Perception. Artificial Intelligence, 167(1-2), 170-205.

Searle, J. (1980) Minds, brains and programs. Behavioral and Brain Sciences 3, 417-457.

Searle, J. (1983) Intentionality. Cambridge: Cambridge University Press. 10

Vogt, P. (2002). The physical symbol grounding problem. Cognitive Systems Research, 3(3), 429–457.

Wagner, K., Reggia, J. A., Uriagereka, J., & Wilkinson, G. S. (2003) Progress in the simulation of emergent communication and language. Adaptive Behavior, 11(1):37--69.

Ziemke, T. (1999). Rethinking Grounding. In A. Riegler, M. Peschl & A. von Stein (Eds.), Understanding Representation in the Cognitive Sciences, pages 177-190. Plenum Press: New York.

TERMS AND DEFINITIONS

Symbol Grounding Problem: the problem related to the requirement of symbols to be grounded in something else then other symbols, if a symbol is to represent something to an artificial system.

Symbol Systems: a system that models intelligent action as symbol manipulation alone.

Sign: something that stands for something else in a certain aspect to someone.

Representation: the same as a sign.

Symbol: a sign that stands for its object by means of a law, rule or disposition.

Icon: a sign that represents its object by means of similarity or resemblance.

Index: a sign spatial-temporally (physically) connected with its object.