A tale of two modalities: How modality shapes language production and visual attention Francie Manhardt © 2021, Francie Manhardt ISBN: 978-94-92910-24-0 Cover design and illustration by Marlijn ter Bekke and Francie Manhardt Printed and bound by Ipskamp Drukkers b.v. The research reported in this dissertation has been supported by NWO VICI Grant awarded to H. A. Özyürek. The educational component of the doctoral training was provided by the International Max Planck Research School (IMPRS) for Language Sciences. The graduate school is a joint initiative between the Max Planck Institute for Psycholinguistics and two partner institutes at Radboud University - the Centre for Language Studies, and the Donders Institute for Brain, Cognition and Behaviour. The IMPRS curriculum, which is funded by the Max Planck Society for the Advancement of Science, ensures that each member receives interdisciplinary training in the language sciences and develops a well-rounded skill set in preparation for fulfilling careers in academia and beyond. A tale of two modalities: How modality shapes language production and visual attention Proefschrift ter verkrijging van de graad van doctor aan de Radboud Universiteit Nijmegen op gezag van de rector magnificus prof. dr. J.H.J.M. van Krieken, volgens besluit van het college van decanen in het openbaar te verdedigen op vrijdag 12 maart 2021 om 14:30 uur precies door Francie Manhardt geboren op 11 april 1991 te Erfurt (Duitsland) Promotor: Prof. dr. H. A. Özyürek Copromotor: Dr. S. M. Brouwer Manuscriptcommissie: Prof. dr. F. Huettig Dr. R. G. Bosworth (Rochester University, Verenigde Staten) Prof. dr. M. Gullberg (Lunds Universitet, Zweden) Prof. dr. C. R. Marshall (University College London, Verenigd Koninkrijk) Prof. dr. P. M. Perniss (Universität zu Köln, Duitsland) Für Oma Flora und Opa Thilo Contents Chapter 1: General introduction 9 Chapter 2: Iconicity in spatial language guides visual attention: A comparison between deaf signers’ and hearing speakers’ eye-gaze patterns during signed and spoken message preparation 48 Chapter 3: A tale of two modalities: Iconic signs and speech influence each other in bimodal bilinguals 98 Chapter 4: Iconicity influences visual attention during signed and spoken message preparation in bimodal bilinguals 127 Chapter 5: Word order preference in sign influences speech in bimodal bilinguals but not vice versa: Evidence from language production and visual attention 147 Chapter 6: General discussion and conclusion 177 Appendices References 209 English summary 233 Nederlandse samenvatting 239 Deutsche Zusammenfassung 247 Acknowledgements 255 Curriculum vitae 261 Author publications 263 MPI Series in Psycholinguistics 265 Chapter 1 General Introduction 10 CHAPTER 1 Language is a powerful tool that enables us to talk about our experiences with the world. However, languages differ fundamentally from one another in how they package and represent different aspects of our experiences. Hence, there is linguistic variation across speakers of different languages as well as among bilinguals that speak different languages (see Evans & Levinson, 2009). This diversity has motivated researchers to explore whether different expressive options across different languages influence how speakers perceive and conceptualise the world differently. In the current thesis, I aim to enhance our understanding of how language diversity can affect cognition for language production by comparing users of a spoken language versus a sign language, focusing on spoken Dutch and Sign Language of the Netherlands (NGT, Nederlandse gebarentaal). In doing so, I zoom into the domain of spatial expressions where spoken and sign languages differ radically from each other due to different affordances of the vocal versus visuo-spatial modality. This enables me to go beyond previous investigations on the effect of language on cognition that have focussed on differences between spoken languages only. Many theories on the relationship between language and cognition suggest that human perception and cognition are universal. Thus, irrespective of which language one speaks, all humans perceive and conceptualise the world in the same way (e.g. Jackendoff, 1996; Pinker, 1995). However, others claim that diversity across languages can influence speakers’ cognition differently, although scholars share different beliefs in terms of which aspects of language and the language production process could influence cognition. Some argue that if language influences cognition, these effects are robust and should extend to domains of cognition outside of the contexts of language use (e.g. Sapir, 1912; Whorf, 1997). However, more recent views argue that language can influence cognition only in the moment of or immediately prior to speaking, which is captured in the so-called thinking for speaking hypothesis (Slobin, 2003). In particular, during message preparation, speakers of different languages attend to different features of the world linked to the way they plan to speak about them. Thus 11 GENERAL INTRODUCTION linguistic categories, which often differ across languages, seem to influence cognition during the process of converting thoughts into language (e.g. Flecken, Carroll, Weimar, & Stutterheim, 2015; Flecken, Von Stutterheim, & Carroll, 2014; Gleitman, January, Nappa, & Trueswell, 2007; Griffin & Bock, 2000; Slobin, 2003). However, this body of research on the effects of linguistic diversity on cognition have exclusively focussed on spoken languages and until now overlooked sign languages, the natural languages of deaf communities. Consequently, it is less known to what extent differences in modality; that is, using the vocal versus visuo-spatial modality, can shape how conceptualization for language production differs between speaking (i.e. thinking for speaking) and signing (i.e. thinking for signing). Furthermore, previous studies have mainly looked at the effects of language diversity on cognition across populations that use different languages but often left out linguistic diversity within populations that speak multiple languages; that is, bilinguals. The current thesis attempts to go beyond the previous literature to contribute to the debate on language and cognition universals and diversity. I introduce differences in language modality as a new angle to study the effects of language diversity on cognition and bilingual diversity. I explore this by comparing diversity between a spoken (Dutch) and a sign language (NGT) across hearing speaking and deaf1 signing populations as well as in hearing bilinguals who can both speak and sign, so-called bimodal bilinguals2. This 1 With the term “deaf” I refer simply to the hearing status and do not intend to make any implications about cultural belongings. 2 With the term “bimodal bilinguals” I follow Emmorey, Borinstein, Thompson, and Gollan (2008) and refer to the hearing individuals born to and raised by deaf parents who can speak and sign. In the literature, they are also referred to as CODAs (children of deaf parents), sign-speech bilinguals or also as hearing native signers (e.g. De Quadros, 2018; Emmorey, Borinstein, Thompson, & Gollan, 2008; Pichler, Lillo-Martin, & Palmer, 2018; Quadros & Lillo-Martin, 2018). They form a special group of bilinguals because they have early exposure to both a spoken and a sign language through uninstructed (i.e. no schooling) exposure to language from early on and are thus highly proficient in two languages of different modalities. For more information see section 4.1 Participants. 12 CHAPTER 1 will provide not only new insights into effects of linguistic diversity on cognition but also reveal novel knowledge about how two languages can influence each other across modalities; that is, between speech and sign, which has not been examined before. To do so, I will focus on the domain of spatial language, specifically, on expressions of spatial relations, such as the pen is to the right of the glass, where previous research has identified clear modality-specific differences between spoken and sign languages in their linguistic patterning (e.g. Özyürek, Zwitserlood, & Perniss, 2010; Emmorey, 2002; Perniss, Zwitserlood, & Özyürek, 2015; Zwitserlood, Perniss, & Özyürek, 2012). Modality differences and spatial language as a domain to study effects of language diversity on cognition Sign languages are considered as fully developed languages that are expressed with the visible parts of the body (Stokoe, 2005) and operate in the visuo-spatial modality using hand, face and body movements for linguistic expressions. Although sign languages are found to share similar linguistic patterning to those used by spoken languages (see e.g. Meier, 2002; Özyürek & Woll, 2019), they also contain modality-specific linguistic aspects (Perniss, Özyürek, & Morgan, 2015). These are most prominent in the expressions of spatial language, such as the pen is to the right of the glass (Figure 1A), which require the identification of the figure object (i.e. small object, e.g. pen) and the ground object (i.e. bigger object, e.g. glass) as well as expressing the spatial relation between them (Talmy, 1985; 2003). In such expressions, modality-specific patterning in sign languages arise, first, due to sign languages’ preferences to use iconic form-to- meaning mappings, and secondly, due to the consistent word order they choose for spatial language expressions (e.g. Emmorey, 2002; Kimmelman, 2012; Perniss et al., 2015). 13 GENERAL INTRODUCTION Sign languages, unlike spoken languages3,