PRAXICON: The Development of a Grounding Resource Katerina Pastra1 Abstract. In this paper, we trace the different manifestations of 2 Grounding Resources grounding resources in a variety of Artificial Intelligence applica- tions and present the POETICON project’s development plans for In practice, grounding resources have been used in AI prototypes that building such resource. In doing so, we introduce the PRAXICON, a span a number of decades and a wide range of application areas, from resource that links natural language and sensorimotor representations the SHRDLU system [59], which verbalized visual changes in a 2D of concepts, with the aim of facilitating multimodal and multimedia blocks scene, to medium translation systems (e.g. automatic sports content integration in cognitive systems. We correlate the envisaged commentators), multimedia presentation systems (e.g. automatic cre- resource with semantic language resources and briefly discuss the ation of illustrated technical manuals) and conversational robots of possibility of interfacing these different types of resources. the new millennium (cf. a related state of the art review of more than 50 such prototypes in [43, 40]). In most of these prototypes, the grounding resources link words denoting entities with corresponding 1 Introduction low-level visual features of objects, and/or the actual images of the objects (e.g. superquadric images or wireframe models or 2D draw- The computational integration of sensorimotor and symbolic repre- ing patterns etc.). The words are then linked to conceptual structures sentations (grounding) and in particular of visual (visual objects and which take the forms of ontologies, e.g. [55], semantic frames or actions) and natural language representations has a long history that schemas, e.g. [34, 51, 8], object or scene domain models, e.g. [7]. could be traced back to the very early days of Artificial Intelligence Prototypes that involve dynamic scenes/motion rather than static ob- (AI). In comparing language and vision and arguing on how the cor- jects, use resources that link words denoting motion with correspond- responding representations are associated, Jackendoff has implied ing motion trajectory information; the words are then linked to con- the use of a kind of integrated, grounding resource in the human ceptual structures denoting events, in the form of event templates, memory for use in both natural language and vision understanding e.g. [11]. In other cases, manually or semi-automatically extracted [23, 24]. He suggested that a resource with both conceptual and vi- geometric scene description data which link object information, their sual information should be built and in particular, one that includes motion trajectory and their name/word is used [1, 20, 32]; this first conceptual structures [22] at the leaf nodes of which concepts are grounding level is correlated to a hierarchy of event models which linked to corresponding 3D models of visual objects (cf. the notion of includes event concepts from general to specific that reflect temporal 3D model catalogue in Marr’s theory of vision [33]). In fact, the idea decomposition, e.g. [21]. In conversational robots, resources that link of building a resource that will link visual and linguistic representa- words with corresponding n-ary tuples of feature-values of visual ob- tions goes back at least to the seventies [3]; it was a reduced version jects and functions that implement motion have been used; the links of Jackendoff’s suggestion and involved the association of manually are then associated to conceptual structures denoting classes of enti- detected image regions (objects) and corresponding image feature ties/events [49, 45]. vectors with words (names for the objects); a similar type of resource In all these cases, the grounding resources used have the following has been suggested recently under the term of multimedia thesaurus characteristics: for use in various multimedia processing tasks [10, 5, 25, 54]. In fact, the idea for such resources has been lurking around for decades, 1. they have been created ad hoc, for the needs of a very limited never explicitly said or described and only partially implemented in prototype that works either in a blocksworld (blobs) or miniworld an ad hoc basis in AI [43, 40]. (very limited and simplified domain); the objects and/or events In this paper, we introduce the notion of a PRAXICON, a resource included in the resource are very simple (e.g. soccer ball, motion that bridges natural language and sensorimotor representations of of the ball along the playfield), the resources themselves have not concepts, with the aim of facilitating multimodal and multimedia been reused in any way; content integration in cognitive systems. In what follows, we present 2. there is no exploration of the association level at which words the “precursors” of grounding resources that have been developed and sensorimotor representations are connected, and no system- for a wide range of AI applications and needs, and elaborate on the atic study on how different representations collaborate in forming notion of the PRAXICON as suggested within the framework of the higher-level concepts; it is intuition rather than a rigorous method- EC-funded research and development project, POETICON. We focus ology that lies behind their creation; on the development plan and the envisaged contents of the PRAXI- 3. the visual and motoric representations used in the resources are CON and discuss the role semantic language resources could play in manually created, i.e. visual abstraction is not done automatically, its development. with the exception of some work on automatic 3D trajectory ex- traction (used in a verbalization task of traffic scenes, [16, 2]), 1 Language Technologies Department, Institute for Language and Speech which is still limiting motion representations to the motion path Processing, Athens, Greece email: [email protected] of a rigid object (as opposed to the fine-grained movements of a human body); sensorimotor representations are monolithic/flat resource should comprise of? feature-value representations. 3 PRAXICON: the POETICON Suggestion Scaling up such resources using automatic mechanisms for associ- ating sensorimotor and symbolic representations and enhancing their Developing an automatically extensible grounding resource is the coverage is highly questionable. In another strand of research that main objective of POETICON, a newly funded EC-FP7 research has been using language games as a method to teach agents primi- project (http://www.poeticon.eu). POETICON explores the poetics tive models of language [53], or how to associate words with objects of everyday life, i.e. the synthesis of sensorimotor representations [26], machine learning methods have been followed. Instead of us- and natural language in everyday human interaction. As shown in ing a manually created, fixed grounding resource, the agent is being the previous section, this is an old problem, one that relates to the endowed with a learning module which allows it to develop such long Artificial Intelligence (AI) quest for meaning. resource on its own through e.g. supervised learning techniques. A However, while meaning extraction and generation with sensori- human feeds the robot with a word which corresponds to the image motor or symbolic representations (i.e. moving or static images, ac- of an object in view; the robot is able to generate a simplified repre- tion, text/speech) has been the objective in most AI sub-disciplines sentation of the image of this object in the form of feature-value vec- (e.g. Natural Language Processing, Image Understanding etc.) the tors standing for the colour and shape attributes of the object. Once research focus has mostly been on individual types of representa- a number of associations are learned, the agent compares the (rep- tions (e.g. visual, motoric or linguistic) rather than on their integra- resentation of the) image of any new object with the ones it knows tion [43, 40]. There are a number of interrelated reasons that have using e.g. a nearest neighbour algorithm and associates the new ob- hindered research to focus on such integration: difficulties in mea- ject with the name of the known object it is more similar to. Similar suring sensorimotor human behaviour, and analyzing visual and mo- statistical or connectionist algorithms for associating words in an ut- toric representations, the tendency for isolation among researchers terance with objects in view have also been developed in a limited working on e.g. extracting meaning from language or images, lack number of artificial agents [48, 39, 46, 57, 44, 6]. of a corresponding theoretical, cognitive or/and neurophysiological This approach in developing a grounding resource has most of background that would provide a solid basis for computational inves- the limitations of the symbolic approaches mentioned above: the re- tigation of the issue. source built by the agent is limited to blobs or simple objects (mo- POETICON suggests a new approach within this AI quest for toric representations are not treated at all), it relies on the developer’s meaning; it captures/measures and computationally structures hu- intuition for choosing the words to be associated with the visual rep- man behaviour (human body movements, facial expressions, manip- resentation of an object, while the developer intervenes in visual ab- ulation of objects) with the objective of creating a PRAXICON, a straction by presenting to the agent a clean image of an object (i.e. grounding resource, and mechanisms that will facilitate its automatic one stripped from any background, or the presence of other objects extension. In particular, the PRAXICON will be the result of the fol- which could lead to wrong associations, etc.). More importantly, the lowing: resource does not connect the words the agent learns with higher- The POETICON corpus: a corpus of four distinct though level conceptual structures (i.e. more complex concepts), which ren- interrelated- sets of multisensory recordings of human movements, ders scaling for the needs of everyday interaction questionable.