Towards Bidirectional Processing Models of Sign Language: A Constructional Approach in Fluid Construction Grammar Remi van Trijp ([email protected]) Sony Computer Science Laboratory Paris 6, Rue Amyot, 75005 Paris France Abstract This paper proposes that a construction grammar architec- ture (Fillmore, 1988; Goldberg, 1995; Croft, 2001; Ostman¨ & Sign languages (SL) require a fundamental rethinking of many basic assumptions about human language processing because Fried, 2004; Steels, 2011) offers a solution to these problems. instead of using linear speech, sign languages coarticulate fa- The most salient property of construction grammar is that it cial expressions, shoulder and hand movements, eye gaze and eliminates the need for separate layers of linguistic organiza- usage of a three-dimensional space. SL researchers have there- fore advocated SL-specific approaches that do not start from tion, such as the sharp distinction between lexicon and syn- the biases of models that were originally developed for vo- tax. Instead, constructions (i.e. conventionalized mappings cal languages. Unfortunately, there are currently no process- between meaning/function and form) are posited as the sole ing models that adequately achieve both language comprehen- sion and formulation, and the SL-specific developments run data structure for representing linguistic knowledge. Con- the risk of becoming alienated from other linguistic research. structions are thus able to simultaneously access and manipu- This paper explores the hypothesis that a construction gram- late multimodal sources of information, which offers unique mar architecture offers a solution to these problems because constructions are able to simultaneously access and manipu- opportunities for modeling sign languages. late information coming from many different sources. This The paper illustrates its claims through a proof-of- claim is illustrated by a proof-of-concept implementation of a concept implementation of a basic grammar for French basic grammar for French Sign Language in Fluid Construc- tion Grammar. Sign Language in Fluid Construction Grammar (FCG; www.fcg-net.org; Steels, 2011), an open-source grammar Keywords: sign language; language processing; construction grammar; computational modeling formalism for exploring constructional analyses in language formulation, comprehension and learning. The goal of this Introduction implementation is not to replace or compete with existing models in sign language research, but rather to demonstrate Sign languages (SL) require a fundamental reassessment of how linguistically sound and bidirectional processing mod- many of the basic assumptions about human language pro- els could work in tandem with the recent SL-specific devel- cessing because instead of using linear streams of speech, opments. The FCG-implementation may thus also lead to sign languages coarticulate facial expressions, shoulder and tighter interactions between sign language research and the hand movements, and eye gaze (Emmorey, 2002). Adequate rest of the linguistic enterprise. models of sign languages must also be able to cope with the use of a three-dimensional signing space and with “the om- Sign Language Modeling nipresence of iconicity at all levels of the language” (Braffort & Filhol, 2011, p. 192). The study of sign languages is Sign languages are so-called under-resourced languages, therefore of utmost importance for our understanding of hu- which means that there are only few reference grammars and man cognition. Moreover, the World Health Organization es- small corpora available (Braffort & Filhol, 2011). Formal timates that over 5% of the world population (360 million sign language modeling thus began in all earnestness using people including 32 million children) has disabling loss of rule-based grammars that have strong ties to particular lin- hearing, so the development of intelligent systems for sign guistic theories, such as the ViSiCast system based on Head- languages has an enormous potential for societietal benefits. Driven Phrase Structure Grammar (Marshall & Safar, 2004). Unfortunately, while lexical signs can now reasonably be Most traditional theories invite SL-researchers to analyze recognized and produced in their base forms, many hard SL utterances as a sequence of lexical gestures that are com- problems remain unsolved when it comes to grammatical pro- bined with each other in a phrase-structural analysis. How- cessing (Vogler & Goldenstein, 2008). One important issue ever, as argued by Braffort and Filhol (2011), all other fea- that SL researchers have identified is that mainstream lin- tures must then be aligned with the boundaries of these lex- guistic theories and contemporary language technologies are ical gestures, which makes it hard to capture coarticulation too strongly biased towards the properties of vocal languages in sign language syntax. Moreover, sign languages are much (particularly Indo-European languages), and have therefore more complex than sequences of signs in their citation form developed SL-specific language technologies (Huenerfauth, (i.e. signs as they appear in isolation): in continuous sign- 2006; Filhol, 2012). Currently, however, there are no models ing, several parameters of a sign can be modified with infi- available that adequately handle both language formulation nite possibilities, as illustrated for the sign BALL in French and comprehension, and there is a risk that these important Sign Language in Figure 1 (from Braffort & Filhol, 2011, Fig. research efforts become alienated from the rest of linguistics. 9.1). This Figure shows the citation form of the sign on the 668 tics that grammatical phenomena can be divided into a core (which can be described as a set of rules) and a periphery (which is a list of exceptions). Construction grammarians also reject the assumption that language is divided into differ- ent, largely independent modules such as phonology, syntax and semantics. Instead, they argue that all linguistic knowl- edge can be described using constructions as the central rep- resentation unit. Charles J. Fillmore, widely recognized as the father of construction grammar, defined a construction as follows: By grammatical construction we mean any syntactic pattern which is assigned to one or more conventional Figure 1: This Figure from Braffort and Filhol (2011, Fig. functions in a language, together with whatever is lin- 9.1) shows the citation form of the sign BALL on the left, guistically conventionalized about its contribution to the and its parametric variation on the right. meaning or the use of structures containing it. (Fillmore, 1988, p. 36, italics added) left, which consists of two hands making a circle movement. I italicized the words any and whatever in this quote to On the right, the parametric variation of the sign is illustrated. emphasize the fact that constructions are able to represent For instance, the speaker can express a big ball by increasing any kind of mapping between meaning/function and form. the radius of the circle (Rad in the Figure), or identify the For instance, as Fillmore (1988, p. 35) explains, construc- spatial location of the ball with respect to a previously signed tions are not limited to the immediate-dominance constraints referent by changing the location of the circle movement (Loc of phrase-structure rules (i.e. every phrase-structure rule is in the Figure). a relation between a parent and its immediate children in a Besides rule-based grammars, the increasing availability local tree configuration) but can make direct reference to the of annotated corpora (e.g. DEGELS1; Braffort & Boutora, linguistic information they require, wherever this information 2012) has now made it possible to also include techniques may be located. from machine learning and probability theory in the develop- Constructional approaches are currently thriving in all ar- ment of sign language models. Most of these methods, how- eas of linguistics (Goldberg, 1995; Croft, 2001; Ostman¨ & ever, have been developed for speech and have only limited Fried, 2004) and have also attracted the attention of formal success when applied to sign languages, particularly when and computational linguists (e.g. Steels, 2004; Bergen & it comes to the spatial properties of SL (Dalle, 2006) or the Chang, 2005; Boas & Sag, 2012). The most advanced line of aforementioned fact that signs rarely occur in their citation work in computational construction grammar was instigated form (Braffort & Boutora, 2012). by Steels (2004), who proposed a computational data struc- Sign language researchers have therefore started to advo- ture in which constructions can be implemented as mappings cate and implement SL-specific methods and technologies. between two feature structures (typically representing mean- One example, discussed in more detail by Filhol (2012) and ing/function on the one hand, and form on the other). This in Braffort and Filhol (2011), is the AZee language for SL gen- turn led to the development of Fluid Construction Grammar eration. The AZee language is fundamentally different from (FCG; Steels, 2011), which is currently the only construction static constraint systems in that it takes a more procedural ap- grammar formalism to handle both language formulation and proach: constraints in the AZee system can best be viewed as comprehension.1 a set of instructions that build XML specifications, which can In recent work, Steels (to appear) has offered a formal then be visualized as sign language utterances using