Restraining and Encouraging the Use of Gestures: Exploring the Effects on Speech

Sede amministrativa: Università degli Studi di Padova Dipartimento di Studi Linguistici e Letterari

CORSO DI DOTTORATO DI RICERCA IN: SCIENZE LINGUISTICHE, FILOLOGICHE E LETTERARIE CICLO XXXII

Coordinatore: Prof. Rocco Coronato Supervisore: Prof.ssa Maria Grazia Busà Co-Supervisore: Prof.ssa Pilar Prieto

Dottoranda: Alice Cravotta

Thesis Speech Barcelona PhD PhD

AliceCravotta , Reggio Calabria Reggio ,

Universitat Pompeu Fabra,

ii Universitat Oberta de Catalunya, Barcelona Università degli Studi di Padova ICREA

Tilburg University 2019 Gibert -

Humanities Lab, Lund University

Università per Stranieri Dante Alighieri Università degli Studi di Napoli Federico II

December Professor, Italy Marc Swerts Full Professor, The Netherlands Maria Graziano Researcher, Sweden Emanuela Campisi Professor, Italy Cutugno Francesco Pilar Prieto Research Professor, Catalonia Núria Esteve Associate Professor, Catalonia Maria Grazia Busà Associate Professor, Italy

di Padova,

Restraining and Encouraging the Use of Gestures: Exploring the Effects on on Effects the Exploring Gestures: of Use the Encouraging and Restraining Committee Members Reviewers Supervisors: Università degli Studi Doctoral School of Linguistic, Philological and Literary Sciences ………… …

) ………… be

aims … do do not speech speech apter 3 how the However, Ch should should ( on

speakers telling obtained the the effects of , speakers were

) tudy tudy 2 not not received much S

This dissertation in

Italian ; ha

hapter 2 C ( speech acoustics mixed results

, research is lacking as to how uction in terms of e.g., speech by exploring re also interdependent at a a in the field.

tudy tudy 1 iii S the process of speech production and restraining and encouraging the use of

aspects Moreover should affect speechshould in similar terms. n

I of . speech prod question Previous research has investigated ; they

both

communicative communicative and prosodic needs. . . However, the

task

by sitting on their hands

on This suggests that impacts debated

ities interact at multiple levels (semantic, pragmatic, effects of as well as acoustic features of speech

narrati Broca’s Broca’s area some of these gesturing to to body movements/gestures in terms of motoric control a

prosodic). prosodic). The underlying mechanisms and functions of this e direct in

rical rical studies were carried out with 40 possible investigat

revious revious studies have shown that speech and gesture production are he restrained from they were encouraged to make gestures while speaking. In both studies, encouraging of gesture use the Two empi short narratives to a listener. inability to gesture fluency, length, and content allow to draw solid conclusions. t gestures on the attention in previous studies connected connected to biomechanical level. interdependent and biomechanics production P controlled by a common motor control system involving a neural network interaction are still a to restraining and encouraging the use of gestures Humans Humans speak and produce hand gestures. It has been shown that hand gestures are a key component that the two modal temporal and Abstract

, , s speech acoustic inability to make use of as as evidenced

and and provide , which also . affect speech

produce produce more , set of s to

by encouraging a

to shown in Chapter (4) that the prosodic prosodic features of as ;

s enact enact characters and (3)

Chapter 4 based on some

iscourse iscourse length in seconds), Moreover, properties (measures of F0 and . potential functions of gesture

pecifically pecifically by analyzing Also, it show iv assessed assessed gesture gesture and prosodic structures are length length and acoustics enhance adopted in theadopted studies

(5) to bodily es the corrections, repetitions, insertions, multimodal cues. multimodal

- can

how acoustic es , , and s

fluent fluent speech production and spoken prosody gesture gesture leads speakers

various enhancing the gesture stream

if and the inability to gesture does not

the methods affect speech urther investigations in this direction would on F

(1)

. by usin the use of ed light spontaneous narrativ - gesture gesture production ), ), speech rate and show that ; ; however, (2)

Additional Additional qualitative and quantitative analyses on the data of the target narratives was for Studies 1 and 2 are reported in

this thesis investigat some aspects of some aspects does not become significantly longer, more disfluent or

is not detrimental to more often in semi

increase in F0 and intensity metrics

results

encouraging

jointly planned and produced. and planned jointly gesture in narrative speech contribute to sh Overall, production in the speech planning and articulation phase evidence that speech gestures gestures and in a higher (more salient) gesture space more representational gestures and actions speakers to gesture can by an 4, The which monotonous interruptions intensity). collected discusses prosodic prosodic and textual features discourse length (number of words and d disfluencies (filled pauses, self the speech ………… …

o ………… o

del i in

l’uso …

quali è

Tuttavia, . l’inibire propone di esplorat (semantic dai per motivi

, per questo,

esto abbia un parlato dimostrato dimostrato che la

sia sia in termini di incoraggiare , del come qu sono stati non non sono stati condotti poco noti e infatti la questione non è stata

lla lla produzione orale e che le motorio caratteristiche caratteristiche acustiche del e d caratteristiche acustiche rlanti abbia influenze sulla loro

Il presente lavoro si . È stato

a volte contrastanti a a dipendenza biomeccanica tra le e contenuto le alla gestualità sia

l’espressione orale e la produzione Tuttavia, i meccanismi sottostanti e

sono ancora D’altra parte,

, , focalizzandosi su come e . ). ). gesti. diversi livelli linguistici

narrativo risultati a

l’uso l’uso della gestualità ibire o incoraggiare i parlanti all’uso della

fanno la lunghezza univoche

prosodic i

proposto che e

discorso inibire e sul i possibili effetti derivanti dall’ e

portato a

parlano

e che vi sia anche un no no un comune sistema di controllo motorio mediato nel

a i su , che hanno analizzato, ad esempio, strettamente legat

nno Questo suggerisce che l’uso l’uso della gestualità nei pa

alcuni alcuni di questi aspetti no questa stretta interazione , temporal

li effetti di o e persone sia G sulla fluenza oggetto oggetto di molti studi sperimentali. uesti uesti studi studi studi sperimental medesimi sui del aspetti parlato. gestualità della studi studi precedenti impatto q difficile trarre conclusion interessante investigare se in gestualità possa avere degli effetti parlato. sulle biomeccanici o riconducibili al controllo esigenze comunicative e prosodiche. Tuttavia, sufficientemente investigata a livello empirico. A questo fine, si ritiene gestuale gestuale condivid dall’area di Broca due modalità. parlato l’incoraggiare produzione orale Studi precedenti hanno le cause di sono investigare Quando Quando l gestualità è una componente fondamentale due modalità interagiscono pragmatic Sommario .

, i - e il

(1) (2) , più che

: sono F0 ; do do le a e può

alcune ad ad es. n

a livello i parlanti istruzion

) mostra divent una una serie di 3 nei quali 40

tali

anche si

arlanti arlanti di usare misure di la la gestualità dei In particolare, si ) delle brevi storie a . Tali analisi fare uso di più gesti apitolo in termini di numero

C spazio più alto ( alcune

più monotono ( (3) In entrambi gli studi

o . strategie strategie multimodali di a:

apitolo 2 chiedere ai p

(che infatti non descritto descritto C

sfluenze (pause piene, auto nel Capitolo 4 che parlato

fluente vi vengono affrontate hanno del sui sui dati raccolti

i Infine, Inoltre, ulteriori analisi qualitative e

meno no i parlanti

). sequenze narrative può avere effetti sulla nel secondo studio ( maggiormente

taliana i le acustic

nel quale , né porta

, Nel primo studio (

ai soggetti è stato chiesto di sedersi tene l’impossibilità di spiegarsi con la gestualità non è elevate (

mentre usare lo lo studio dimostra

che più

) della voce. ono ono stati condotti due studi sperimentali

o di durata (4) n

velocità velocità del parlato (sillabe al secondo), e la la lunghezza delle produzioni orali ( inibita , s i

madrelingua madrelingua i parametri ;

ono ono state condotte Capitolo 4 la coraggiati coraggiati all’uso della gestualità il numero di gesti che usano nel raccontare le storie mostra D’altro lato risultano

stato stato analizzato a livello prosodico e testuale.

le stesse istruzioni ) o in termin questo lavoro risultati Inoltre, rappresentativi aumentare gesticolare in modo più “saliente”, ovvero in uno in corrispondenza del petto) rispetto a quando non ricevono lunghezza lunghezza delle storie narrate (in termini di numero interagire di con parole) intensità incoraggiare i parlanti a usare la gestualità li porta effettivamente a lungo acustic la gestualità nel descrivere questioni metodologiche. questioni I di particolare impatto negativo sul parlato misure acustiche che riguardano la l’intensità (volume frequenza fondamentale (F0) e quantitative s descritte nel di parole e lunghezza in secondi), correzioni, le ripetizioni, di inserimenti di interruzioni), suoni o segmenti di parole, mani sotto le gambe), sono stati in parlato è sono analizzate: partecipanti partecipanti di fumetti a un ascoltatore. parlanti è stata In ………… … ………… … alcune insieme ulteriori funzione

futuro,

particolare nella

lla possibilità che In possibile suggeriscono che la

spontaneo. - e delle spalle.

risultati

e. I

di investigare la .

vii piegarsi tramite la gestualità può essere propone e articolazione movimenti della testa movimenti della processo processo di espressione verbale, in

(impersonificazione) che coinvolgono anche, ad esempio, processo diprocesso produzione verbal pianificazione durante il durante mostrano mostrano che l’impossibilità di s ben compensata nel discorso narrativo semi studi sperimentali potrebbero contribuire a fare luce su gestualità e strutture prosodiche siano pianificate e prodotte sua produzione della gestualità caratteristiche può prosodiche contribuire del parlato. a Allo enfatizzare stesso tempo, i risultati espressioni facciali In generale, questa tesi si della gestualità nel enactment enactment

65 67 25 32 39 41 42 43 43 47 50 15 17 22

61 27 31 42 52 53 11 13 17

59 29 11

......

...... viii ......

......

tions, fluency and acoustics and fluency tions, ......

......

...... speech gestures: Types and minimal units

- ......

Results Discussion and Conclusions Introduction Introduction Method Gesture and Speech Production Aim of the thesis What is gesture Co

Previous studies on encouraging gestures & researchSummary question Encouraging the Use of Gestures: Effects on Speech Effectson Gestures: of Use the Encouraging Materials Procedure Data Analysis: Transcrip Statistical analysis Influence of gesture production on speech acoustics speech on production gesture of Influence & research Summary question Participants Exploring the Effects of Restraining the Use of Gestures on Speech on Gestures of Use the Restraining Effectsof the Exploring Previous studies on restraining gestures Speech and gesture production models Gesture and speech integration: Temporal alignment and acoustics Gesture and disfluent speech Gesture temporal structure Theoretical Background Theoretical 3.1 2.3 2.4 2.2 2.1 1.4 1.1 1.2 1.3

Introduction Contents ………… …

5 ………… 94 95 96 98 98 99 68 68 69 70 73 73 88

94 97 68 75 77 84 86

83 93 .

119 122

119

101 119

...... 12 ...... … ......

......

...... ix ......

......

...... use of gestures d Study 2

...... the ......

......

Encouraging Encouraging gestures Restraining the use of gestures Results Discussion and Conclusion Restraining gestures Method

Study 1: Restraining Gestures Study 2: Encouraging Gestures Fluency Prosodic features Fluency Prosodic features Speech length Effects of encouraging gestures on gesture production Conclusions and Discussion General Speech length Restraning and Encouraging Gestures: Methodological Remarks Methodological Gestures: Encouraging and Restraning Statistical analysis Participants Materials Procedure Assessment of the effects of the encouraging gesture prompt Data analysis: Transcriptions, fluency and acoustics Materials for Study 1 an 5.2 5.1 4.1 4.2 3.3 3.4 3.2

Acknowledgements…………………………………………………………… Appendix References

73 74 76 16 43 44 71 71 72 77 89 91 37 51 52

......

...... (N, E) (N, ......

(N, E) (N, ......

......

...... Salient gesture rate (NS) - iment ...... representational gesture rate rate gesture representational ...... -

......

salient gesture (right) salient gesture x - ......

ords)......

......

scene comic stripfor exper the comic used scene - effects models for the effects of condition on Gesture rate, Salient gesture - ......

Salient gesture rate (per w 100 - up. -

Table 2 Main descriptive statistics Table 3 Results of the per variableLMEMs dependent Table 4 Linear mixed Non and rate Table 5 Main descriptive statistics Table 6 Results of the per variableLMEMs dependent Figure Figure 8 Box plotsNon and reporting Representational Figure of 9 Examples enactments.. Tableof 1 Summary previous findings on the effects of restraining gestures on speech. Figure 2 Example Figure of 2 a Example 4 Figure 3 Set on McNeill, (based 1992) representation Figure space 4 Gesture Figure of 5 a Examples salient (left)gesture non and Figure rate 6 (G) SalientGesture and rate (S)gesture Non and Figure 7 a Box plotsthe significant variables that representing showed effect of Condition (N, E) Figures and Tables Figures and Figure of 1 a Example phrase. short gesture ………… … …………

). ). … In 12 - ,

: Effects (2019), (2019), ). ). G.

2019 under review ( M.

al Gestures. In , , Paderborn, 11 ) M. Busà Busà

, N & eSpI

. based on thebased on following G

Conference on Gesture and is 26

G., & Prieto, P. ( -

th xi 21

. pp. Prieto, P.,

Effects of Restraining the Use of Gestures , , Busà, M. , , Prieto, P., & Busà 3219 dissertation dissertation

. 3204 in this

s Encouraging Encouraging Gesture Use in a Narration Task Increases Speakers’ Gesture Production Rate, Gesture of Proceedings Salience of the Representation and 6 Speech in Interaction ( September 2019 Cravotta, A. of Encouraging the Use of Journal Gestures on of Speech Speech, Language, and 62(9), Hearing Research Cravotta, A., Cravotta, A. Exploring the Speechon

Chapter 4 Chapter 3 Chapter 2

List of Publications of thePart content

xii

………… …

………… as e.g.,

( …

kan & ş Bavelas, Bavelas, the italian

Koenderman, e.g., - (

This suggests that post by . Introduction he image was captioned like people with no visual ow, ow, 1998) congenitally congenitally blind individuals an Instagram just just Mead

- and and

thinking and speaking processes Meadow, Meadow, 2015; Church, Alibali, & Kelly, - the the words would choke in my throat. If I didn’t

Mol, Mol, Krahmer, & van de Sandt and and break down together in patients with I came across

e.g., Gestures Gestures develop together with speech in children gesture ( . Iverson Iverson & Goldin 4; 4; Gullberg, De Bot, & Volterra, 2008; Özçali ing and gesturing, and t e.g., (

people gesture while talking over the phone

have some role in the

(EN) (EN) If I didn't

Meadow, Meadow, 2005) - Graziano, Graziano, 201 . . Before discussing gesture in more detail, let us see what is . Also, e.g., follows: At some point last year, journalist and writer Roberto Saviano. The journalist was portrayed in a photo while speak gestures Abner, Cooperrider, & Goldin 2017) gestures. known about commonly Gerwing, Sutton, & Prevost, 2008) gesture while talking to each other impairment do ( Goldin language impairments 2013) Humans Humans speak and produce hand gestures. These spontaneous movements that accompany speech across languages and cultures are a pervasive part of all human language

’s found

trivial,

it, many of hat hat I individual

. 1 an

gesture when we us. We can’t do gesticolassi non riuscirei ad it’s it’s very difficult not

, Can we even talk with your hands.

ded while speaking: may may be considered comments below

about why we (Roberto (Roberto Saviano) to to the heart of the interlocutor. If I 280 makes the words body and color. Once a surprised friend I was of talking hands in the pockets… with mine the was Even for me me! of It’s part gesture. to It’s stronger than without it! Seriously. without gesturing at least a little? Ah ah! I’ve always told speak you you this: A language in the language that

around around

. . It was interesting (and fun) to see how these comments those those words

commonplace and can hardly be hardly avoi can and

Though .

necessary I would speak only halfway

I couldn’t accompany stupita perché parlavo tenendo le mani in tasca… Anche per me è difficilissimo non gesticolare. è parte di me! è più forte di noi, non riusciamo a meno! a farne con lecon mani. Una lingua nella lingua che rende le parole corpo e colore. Una volta un'amica si è mostrata Ma perché, sul serio si può parlare po’? un gesticolaresenza manco Ah ah! te l’ho sempre detto! Parli and and what is gesture for

gesture

Se non gesticolassi le parole mi si strozzerebbero in gola. Se non

accompagnarle sino al cuore dell’interlocutore. Se non gesticolassi parlerei solo a metà (Roberto Saviano, September 2018).

personality personality and style and Naples) Italy, (e.g., background also linked to one’s cultural/linguistic Also, Also, gestures are intuitively believed to be part of speak people feel about it they suggest a few intuitive things about gesture; for example, gestures are considered didn’t interesting was the presence of them being some sort of gesture 1 ………… …

: ………… … voice

impact on

of their

sound an emotion,

more effective …

s e and life. They arrive

. z e “cultural gesturers”. hts To the heart, to the municative gesticulation! True! They thoug expand your Do gesture! It better explains what you say. It underlines emphasi a a concept! strengthens thought, If you didn’t gesture I wouldn’t believe you Gesticulation is the echo of the words, which louder in the heart reverberate who watch and listen to you. of those It is with gestures that words take shap exactly where they are meant to. stomach. If I don’t hands I’m not myself. speak moving my If you didn’t wouldn’t be Italian… gesticulate you For us, gestures come Neapolitans, words because before the the concept is expressed the first with the hands. We ar Yours is an com expressive and

that that gestures are also commonly believed

”.

concepts concepts better, and to have a

… gesticolanti culturali “

esattamente dove arrivare; al cuore, alla devono allo stomaco. pancia, Vero! è una forma vera di e estensione propria dei tuoi Se tu non gesticolassi, io non ti crederei Gesticolare è l'eco delle parole, piú forte nel cuore rimbombano di chi ti osserva e ascolta. È che con prendono i forma, vita. Arrivano gesti le parole È un gesticolare espressivo comunicativo! e Gesticola! Rende meglio ciò che dici. Sottolinea il pensiero, enfatizza un'emozione, concetto! rafforza un italiano... Per noi vengono napoletani prima i poiché delle gesti parole il espresso prima con concetto Siamo le mani. viene Se non parlo muovendo le mani non sono io stessa. Se non gesticolassi non saresti express Some people also referred to the fact that gesture can have effects on the pair well with also thespeakers can and themselves, Some other comments show to aid to the listener:

, - As . solving - (Goldin thinking

Precisely, 2017) s show that

. seem to limit the idea that

they

the human need

of language and

in … (Chu & Kita, 2011, reflect the thinking

make words their function being

s part to to be more profound: hey

(McNeill, 2005, melody, a counterpoint - seem

Italian contexts, communicative, problem - of language, can can be found -

It's It's a way stone ... it's a way to feel, shape, to sculpt words, sculptor. It is so for me. just like a I’m like you. If I didn’t gesture I would feel speechless Gestures are a second voice, a counter perhaps? to the musicality voice.your of

Meadow, Meadow, 2010; Chu & Kita, 2011; most of the comments - thought gesture 4

- and and ornament speech. However, the co

, for this …

(Kendon, (Kendon, 2008) that that underlies speaking

anto, un with speech” (p. 78). In fact, when people are not ” ntative of other non on - e, integrate z (Beilock & Goldin

imagery add

. “even if, for some reason, the hands are restrained and a

parole, proprio scultore. È cosi me. per come uno Sono come gesticolassi mi sentirei muta te, se non I gesti sono una seconda voce, un contrappunto controc musicalità della tua voce. forse? alla pensieri È un modo per rendere pietra le parole... è sentire, un modellare, modo scolpire le per , , they can still produce Possible explanations . mpirical findings have shown that gesture production influence e problem solving 2016) gestures are capable to support more general cognitive processes Meadow & Alibali, 2013; Hostetter & Alibali, 2019; Kita, Alibali, & Chu, 2017; Pouw, de Nooijer, van Gog, Zwaan, & Paas, 2014) hidden but integrated speaking, but instead are in silent, non situations process process and the McNeill (2017) puts it, gestures must be considered for example, gesture is not externalized, the imagery it embodies can still be present, only to emphasi integration between gesture and speech gestures are inseparable from language because t people have some general ideas about gestures as part of to communicate. At the same time, gestures to an “ Though all these comments are drawn from the Italian context and may not be fully represe ………… …

- ………… in the

… Goldin scholars

(Campisi & have a rich .

the creation of ; (Broaders, (Broaders, Cook, and and language

but also

(Cook, (Cook, Yip, & intentional Arbib, 2012; Armstrong &

. As a consequence, people ; bond between gestures and , learning

as a potential starting point for memory memory

be explored in the evolutionary ight

and and

5 also ddressees, goals, and contexts; second, , and and contexts is The t

, . common common developmental origins of the two cognitive cognitive functions

evelopment of gesture Meadow, Meadow, Nusbaum, Kelly, & Wagner, 2001; - and and can Abner et al., 2015)

the d

has been proposed ; proposed . In fact, they integrate with language at all levels of

e.g., ( Meadow, Meadow, 2007) - dimension as well. However, disentangling Gesture Gesture First Hypothesis gestural forms from spontaneous gestures; the ( efficient way for a - esture

Meadow, Meadow, 2010) .

- (i.e., (i.e., communicative intention) is not an easy task. First, this

, gestures serve rich the decay of gesture in language impairments ; human cognition children codified/shared between gestures and speech in terms of semantics, pragmatics, syntax, phonology, temporal alignment; the role of gesture in social interaction and Mazzone, 2016) Mazzone, Over the past decades, these questions have gathered together from different disciplines that have studied, for example, the relationship communicatively it would also require an understanding of adaptation to the addressees, extent goals to which such communicative communicative functions of gestures and intentionality reconstructing the role of would require an understanding of whether gestures are designed in a linguistic structure cannot do without gesturing when thinking and speaking. Also, together with their cognitive and linguistic dimensions, gestures have a Wilcox, 2007; Corballis, 2003; Stokoe, 2001; Tomasello, 2008) Tomasello, 2001; Stokoe, 2003; Corballis, 2007; Wilcox, In essence linguistic dimension speech may lie on the (Iverson & Thelen, 1999) context, where g human language Mitchell, Mitchell, & Goldin Meadow, 2010, 2012; Goldin Ping & Goldin Pouw, Mavilidi, van Gog, & Paas, 2016)

, - s s ’s to

on co

(Novack (Novack different

In essence, . them while

s ions/purpose focuses , computational

has has led to the speech therapy, What makes us

: dow, dow, 2017)

models. and and cognitive processes are critically related and cause

) Mea and and has highlighted some - linguistics, ese topics on on producers and observers )

What types of events make

has historically required the role of gesture in language Church, Church, Alibali, & Kelly (2017) 1.2 ; production production

effects 6 the why rather than the how and and , gesture

1.1 Novack Novack & Goldin e interest on th . . Speculating about possible funct unctions of gesture e.g., unction unction and mechanism its purpose ( f that that is, exploring their ections

gesture arise from? S ? In this wide context, this dissertation ,

es understanding

(see

ollowing the reasoning in F to occur

ere do research. developmental developmental psychology and ). ). way way to gain further understanding of gesture is to focus on the h

of gesturing,

Meadow, Meadow, 2017) W

- 1.3 mechanisms

speaking Section Section

both perspectives (e.g., f are needed for the development of comprehensive gesture gesture? gesture likely & Goldin for gesture requires exploring their and also accounting for these effects in cognitive, neural and social terms. The second milestone book, approach the question from two perspectives. The first is to examine the functions Previous research has contributed a clearer idea of how people gesture when patterns of gesture use in language ( speaking. Mechanisms and linguistics, neuroscience, primatology, human communication studies multimodal speech gestures and their role for the person who produce origins and evolution. Th flourishing of experimental studies that have had an impact on fields such as cognitive science, psychology, psycholinguistics, cognitive relationship between gesture and signs ………… …

it …………

has as it

… search researchers ntrast, two

gain further ,

encouraging Precisely, of gestures on

the presence of to . .

ing . . By co did not find effects of effects

Moreover, re ). ). e expect the Hoetjes, Krahmer, & Swerts,

1.3 ous ous methodologies and tasks manipulat studies studies have investigated the e

, , & Kita, 2007) n n speech fluency, content, and by e.g., o and and low codability abstract lines (

explore have been found in visual objects

7 nteraction nteraction between speech and gesture or example, some detrimental effects of , previous F

is to . has been their use

2.1 fluency . . By doing so, w Hostetter Hostetter et al., 2007) used used heterogen Bernardis & Gentilucci, 2006; Marstaller & Pouw, Harrison, & Dixon, 2019) & Harrison, Dixon, Pouw, (Graham (Graham & Heywood, 1975) ection have

S e.g., the dynamic i e.g., (

( preventing and and that the two modalities can be interdependent at a mixed results restraining gestures

ed estraining and encouraging the use of Morsella Morsella & Krauss, 2004) either explore

. R the role of gesture production for speech fluency and acoustic of this dissertation yield by

anical level anical , These studies s effects of

. Chu & Kita, 2011; Hostetter, Alibali (Hoetjes et al., 2014; aim Broca’s Broca’s area xplain xplain the mechanisms of gesture and speech production tasks restraining gestures on fluency nor speech length. In studies using story description ( drawings descriptions studies which elicited speech by asking participants to describe motor length and have the inability to gesture on 2014; 2014; Rimé, Schiaratura, Hupet, & Ghysselinckx, 1984) (e.g., will be reviewed in direct One way to during thinking and speaking gesture the Burianová, 2015) biomech have developed psycholinguistic models that integrate the gesture stream in the process of speech production (Section shown that speech and gesture production common are motor control both system engaging a controlled neural network by connected to a insight on modulation To e gestures The narrative speech production Goals:

of in

little clear could could ection S the two gestures types

, , second acoustic

Meadow, Meadow,

- Fuente, Fuente, & - its how it received

use of

or retelling tasks

, iscussed iscussed in d possible to draw solid

heterogeneous should should affect speech on our view is that thus far

not question of

encouraging the use of Pouw, Harrison, & Dixon, it is hard to draw

(Beilock & Goldin

production. , has

it is

(Broaders et al., 2007) González, González encouraging the , as it will be -

from the have been overlooked. Precisely, ,

. speech

lso, 8 A

. speech the effects of

However

d . different studies have been found on both fluency and speech (Baills, (Baills, Suárez

, learning math Finlayson Finlayson et al., 2003; Jenkins et al., 2018; acoustic features From these studies explore the inability to gesture impacts speech differently .

Krahmer & Swerts, 2007; Coromina, Coromina, Prieto, & Rohrer, 2018) - spontaneous the results -

Hoetjes et al., 2014) Hoetjes et al., that

fluency, fluency, content, and length and als direct effects of actively Moreover, other possible effects of the inability to gesture . erhaps, /semi P ; Llanes

(but (but see ies . 9 unclear lack lack of studies on how the instruction

proposed in the differently with thinking and speaking), , on activities such as problem solving possible a

speech on s

i . Giménez & Prieto, 2018) age pronunciation

- the scher et al., 1996) connected , the potential functions of gestures for of gestures functions the potential restraining their use (i.e., both conditions can lead to side effects that can interfere types of instructions can in principle provide complementary evidence on properties 2019) Though encouraging the use of gesture is not exactly the polar opposite of on there is levels such as langu Prieto, 201 (Vilà 3.1 previous studies gesture 2010; Chu & Kita, 2011) affect see (but attention On the other hand, participants have been instructed to gesture depending on the different tasks analys commonalit on speech can be explored; for example, the content content and planning ( Rau conclusions retellings ………… …

. : ………… are and and two … ) effects 2 The ture ture and (without (without proposes

that have . integration corrections, - t. These are and and ring the ogical remarks. speech way to investigate speech

on Additional Additional qualitative

and and definition of ges full functioning the results

iction iction (Chapter

of the eech.

restr gesture

. two two empirical studies (described in

spontaneous 9 including two two twin studies explo of

n ecologically valid ) standing standing contributions with their own the use of gestures a issertation - fundamental fundamental frequency and intensity

1 in on narrative sp

)

3 hapter C , , together with some methodol (

4 that that are self that that revolve around specific “modules” of the process, e.g., memory, encouraging

) ). ). The analysis of speech is based on features . 3 (Chapter and and

and and

is devoted to a general discussion

2 ) 5

s restraining apter peech discourse length, fluency (number of filled pauses, self some open questions for research future some open questions and and quantitative analyses on the data collected for Study reported 1 and in 2 Chapter Chapter abstract, abstract, introduction, methods, results and discussion sections. The two studies explore the effects encouragement of gesture theoretical background the main questions The dissertation is then composed Ch measures Outline of the present d This dissertation starts with a chapter that is devoted to the general s repetitions, insertions, interruptions, silent pauses, speech rate), as well as acoustic features of speech ( taxing/challenging lexical access, etc been previously investigated in relation to hand gesture restriction and only marginally in relation to hand gesture encouragemen of studies elicit narrative speech the speech production process in its The present dissertation reports on

………… …

, ………… ve

core

…

(McNeill, The

. properties fully conventionalized sign sign language absent speech linguistic n general, they

answer these questions (Kendon, (Kendon, 1997; McNeill, and and neuroscientists ha

. Kendon’s Kendon’s continuum properties partly conventionalized emblems emblems present speech or absent linguisticsome

>> assified in many ways

Theoretical Background Theoretical . . This broad definition can be further specified

linguists, psychologists

see Abner et al., 2015) pantomimes absent speech no linguistic properties not conventionalized

people gesture while speaking

>> classifies gesture as follows, depending on different why

What is gesture is What

(Kendon, (Kendon, 2004)

addressed by and and

e detailed categorizations. The 1.1 for a review,

how ;

not conventionalized no no linguistic properties gesticulation present speech estures estures have been defined and cl dimensions: can can be intended as exchanges body movements produced in in mor communication 1992, 2005) been 1992 g The field of gesture studies flourished questions in the last decades

, ). e.g., sign 1.2 ( is the

speech mano mano a - gesture.

From the produced produced emblems or

. they have concurrent s . It mainly ions can be presence presence of )

twist - , or often, for there are

, (McNeill, 1992 see Section , p. 5 pistol because present in co Gesticulation embodies “meaning

hand hand purse of use, synonyms,

, , standardization and

(Poggi, (Poggi, 1983) not s the obligatory and and it are conventionalized hand and and it stands for “nothing”

increases

andshape, andshape, movement, location,

connected connected with the speech speech gestures context - that are question (McNeill, 2005

is the Italian co Poggi, Poggi, 2002)

are

s is a (sequence of) gesture

12 speaking (e.g., (e.g.,

elements are the main building blocks of the formedness, of the continuum Kendon, Kendon, 1992) - hand hand movement , pantomime

, elements such as h a

manual - of the

of well signs and and “glossed”

. . On the rightmost side of the continuum s

and and rotated back and forth

As for can can be also referred to as rule amous example of emblem .

ages They , , that can be glossed as a gesture of u out speech that can convey a whole narrative without being . g ) lan orientation, and non It It is performed with the hand having thumb and forefinger extended at right angles, (Kendon, 1992) grammatical grammatical and pragmatic classification gesture. A f borsa Italian repository, a second example can be given: the thumb up gesture that often stands for OK). Munari, They can 1963) be listed meaning, with conventionalized or having any linguistic properties. Next to it, (or “quotable gestures” the (e.g., whose movements community is meaning shared (linguistic) in a brevity, simply as gestures. Gestures speech at the semantic, pragmatic, and discourse levels ( Along the continuum, involves involves hands and arms movements, but it is not limited to these body parts (shoulders, head, face, legs can be part of it). Gesticulat regarded as “symbols of action, movement and space” p.2 most most pervasive type of gesture, spontaneously in in that the it context of is any relatable to motion the accompanying produced speech” Moving Moving from left to right speech decreases and the stability of the meaning linguistic properties ………… …

. - ………… as

and and (like

along … influential dissertation, dissertation, Importantly, Importantly,

adaptors - be described how gestures can

self pragmatic functions of can

s e present 2004; 2004; McNeill, 2005) to their semantics, shape Bavelas, Bavelas, Chovil, Lawrie, & ypes and minimal In th such as ent ent times (first century),

. T gesture overview : e.g., ( (Kendon, (Kendon, s

grained description of gestures as will refer to gesticulations or co -

In anci hand hand

will focus on the most 13 ). .

movements movements a general

etc es esture gesture and

and and have referred their form be grouped, classified and described exclud

will give on on different levels (e.g., the extent to which they

uted uted a more fine n can ed

(book XI, chapter 3). In modern times, different mention of

s speech g

c gestures are imagistically relatable to the ongoing -

xamin esture Co units Iconi next sectio

. and and e to to their description a whole chapter of the rhetoric compendium when they depict images of concrete entities or actions, via the

The 1.2 s . ed temporal temporal coordination with speech speech g - he Kendon’s continuum shape of the gesture (hand shape) or manner of direction). execution (trajectory, and and For example, in terms of iconic Co different dimensions, based on their form, semantics, proposals and and communicative/pragmatic functions Wade, 1992; De Iorio, 1832; Efron, 1941; Ekman & Friesen, 1969; McNeill, 1992) be classified and described Institutio Institutio Oratoria proposals have contrib communicative movements are iconic or carry symbolic meaning, their semantic level properties, their of conventionalization Quintilian described gestures as important devot oratory components speech gesture. Gestures have been observed and described in various decades, ways over the t scratching one’s nose or touching one’s hair) from now on, any sign language phonological and morphosyntactic structure.

. . . - - or or

the on

non ; also, deixis domain domain gestures or - concrete

(Kendon, (Kendon,

etaphoric M In fact, n

cross . or making a

, relatable to via with the hands nteractive to refer to “past” of gesture is abstract pointings i referential

- s (Graziano, (Graziano, 2014) can can be referred to as back non

ing” action imagistically

. movements n engage the cognitive process of taking taking exchanges and dialogic occurring occurring speech but instead are - “climb For example, pointing - (Abner et al., 2015) appear in a great variety of shapes For example, space can be used

abstract in terms of the . ; or express speakers’ stance,

. 14 be no children’s pragmatic development of can

08) semiotic component gestures) gestures) can be performed with any body

functions abstract pointings discourse discourse connectives (McNeill, 1992) . They -

depict the can also these can be referred to as or rather they

; the discourse making making climbing

and and possible )

to several gestures gestures with pragmatic functions can be used by Meadow, Meadow, 2018) s -

in or pointing Bavelas, Bavelas, Chovil, Coates, & Roe, 1995; Cooperrider, gestures

estures ( gestures ensure turn These can be referred to as

the content of the co

. have , , p. 14

e.g., Kita, Kita, 2003) . . Such ne of the ( ing something in terms of something else hand hand (Cienki & Müller, 20 an an open hand over the shoulder to refer to “yesterday”) gestures (see represent though though objects or entities pointed at are often not physically

, metaphoric . . For example, s , conic discourse discourse and use of meta gestures can pave the way to parsing/punctuational parsing/punctuational and marking focus and information status 2004, 2017) children too, and develop together with the latter’s ability to structure useful to frame it representational interaction Abner, & Goldin representational and forms and do not representational gestures representational However, ongoing speech part, part, and are used to indicate a certain direction, location or object in the space present I moving Moreover, o Deictic (McNeill, 1992 understand mapping metaphorically to represent time (e.g., fingers when expressing verbally a rounded shape with the two hands to describe a “tuna can” gestures are those that speech ………… …

- ………… the and and … kinetic (Loehr,

type of

phase of temporal rhythmic d around

constitute either

underlying underlying

changes of

, , including any starts with a

, start start departing pragmatic It is the which

that is, the have both 1980)

( mical pulse

predominant

h . (Efron, (Efron, 1941; Ekman & However, The exact point in time can

(or (or gesture peak) he hands .

t . )

rhyt unfold unfold by passing through a more

Kendon apex batons all gesture types

, , p. 41 , or

. These phases are organize ‘peak of the peak’, communicative role. 15

t upon the express a range of s , , apexes can be identified as (Hübscher, 2018; Hübscher, Garufi, & isely, ’s gestures gestures such as flicks of the hands or

it (Kendon, (Kendon, 1980)

tructure described by and Prec speech gestures Prieto, Cravotta, Kushch, Rohrer, & Vilà s -

as s . representational ones,

Hufnagel &Hufnagel Prieto, 2019) - , being buil - e.g., (McNeill, 2005 down movements have been variously defined as ( - Loehr (2012) can can be referred to as (McNeill, 2005)

. In the preparation phase, The stroke is the part of the gesture that carries the and emporal - gesture . t (Kendon, (Kendon, 1980, 2004; McNeill, 2005) , some types of l since they seem to have a

function phase .

phase anatomy (Tuite, (Tuite, 1993) relevant relevant dynamic points within the stroke (e.g.,

forth/up - d According According to Gesture (Kendon, (Kendon, 2004; McNeill, 2005) . basic

an prototypica other

- stroke

Any preparation moments moments in which the arms/hands maintain their maximum extension or any direction). manifested with greatest clarity in which the movement reaches ‘goal’ of the stroke 2012) the meaning and takes on the gesture the excursion in which the movement dynamics of effort and shape are It It has been observed that co series of phases their interactive functions Shattuck Giménez, 2018; gesture can in principle speaking representational and non and prosodic component beats Friesen, 1969) highlighting Prieto, 2019) In the literature back knowledge state and politeness

of and and data data

return

gesture

either a ee a una Figure a bowl

frame

. then of

aand include central the stroke can be

occurring with the , - stroke holds phases - included included in the can

The text reported above . gesture gesture holds (i.e., the The hands can gesture gesture phrase

video speech co and and post a

. . Holds

- ) appears in the phase). There might not be a . . For example

the phase.

di cibo]

The first picture shows the start time of the from a

stroke food] for example, e stroke

constitute , ciotola the retraction (

delimits the speech accompanying specifically the the end of the retraction phase retraction the of end the

d include again to reach

bold followed by two rest positions. n

, square square brackets enclose

. positio a short gesture phras s a cat with a guy in front of a box that empty [… seems ’

s . The Example of a short gesture phrase rest

phase 1 each other

. Together these phases a interrupted preparation or any pre

/ (EN) There (EN) (IT) C’è un gatto con il suo padrone davanti a una scatola che sembra vuota [… [… vuota sembra che scatola una a davanti padrone il suo con gatto un C’è (IT) preparation phase, the second picture shows the stroke, and the third one represents Figure (Kita, (Kita, van Gijn, & van der Hulst, 1997) This is a simple example, as gesture phrases can be longer and combined with momentary suspension of a movement held appeared three images, and stroke represent collected for this dissertation. The it is preceded and the figure is a transcription of the speech context in which the to a rest position retraction phase if the speaker moves directly from a stroke stroke to a new from ………… …

- - s

2 ………… the

have since …

, gesture gesture are co

that give The next

shown Chapters

the next section framework, that

specifically

occurrence between - they have the same Also,

increasingly

) and that (3) co

. of gestures for speech

e ), ), (2) speech and gesture models levels

, th

Production

for the speaker’

1992)

disfluent disfluent speech expressive speech is over (i.e., post 17 l ( different - These three levels of synchrony the two studies presented in

and and ). ). stroke hold can occur when the stroke is on - e

for occurring occurring gestures and speech relate to the how and why production production of speech and gestures has also - -

McNeil ‘blueprint ous ’s by pragmatic synchrony rule overview of the main theoretical accounts occur in proximity of the most prominent syllable

(1989) synchron ) n

semantic semantic synchrony rule a es

nations as to are the temporal interaction and coordination between proposed prosodic prosodic features

f The dynamic co Gesture and Speech and Gesture Levelt

expla the proposed cognitive models accounting for gesture production

(or apex

1.3 Speech and gesture production of

s upon .

the focus of many empirical have theof many that focus studies

3 originally ld ld in place until the related co phonological synchrony rul draw explains the process of speech production from conceptualization to overt Many these two aspects are relevant and produced produced and will zoom into some empirical findings interact with on how gestures temporally been explored to better understand production the and their role communicative and cognitive functions. section provides potential been complexity o and speech same idea unit ( pragmatic functions ( stroke ( As gesture strokes and certain speech segments suggests that the two relate at least at three levels: (1) co stroke hold). Conversely, a pre ready to be completed but it “waits” for its related linguistic segments to be uttered. he

: - a ) of

each

ing in the

. morpho When a interact. tak converts (

The whole

the building ensures the articulation phonological

- f

with semantic starts.

(construction aimed at n this phase, signal). i

: short overview of the

grammatical grammatical encoding encoding conceptualizer consists o

ls, a lexical concept syllabified words, phrases the (articulation phase) (articulation the input to the formulator

, honetic p encoded ( distinguishes three stages preverbal message

s s and models will be given below e

the formulation phase then of the utterance

articulators activates articulatory gestures from the Kita & Özyürek, 2003; Rauscher et al., is conceptualization rocedural rocedural knowledge through working Overt Overt speech results from the execution

where and how the two streams resulting resulting in auditory .

honological honological encoding (e.g., (e.g., speech

. from p of the utterance i.e.,

s. s. Finally,

articulatory articulatory gestures are generated and triggered, (preverbal message is

debate is grammar, grammar, phonology, lexicon), and process: process:

The p

is where the gesture hypothes . a a corresponding to the - articulated )

in conceptualization phase

articulatory articulatory score This lemm the esture stream is proposed to be integrated into this process. .

articulation phase overtly in the theoretical he preverbal message

, encoded formulation T inspired inspired models Here is where

. the study presented in this dissertation is not is carried out by different modules interacting with each other.

final of the phonological phonological score a new g articulatory score by score articulatory

of all

Levelt the matter of

reverbal lexical concepts activate the corresponding lemmas from the as background information background as Though particular stance towards one or the other mode main existing speech In 1996) A syllable in the phonological score syllabary. for the of of the and intonation pattern generation of mental mental lexicon lemma is selected, the speaker gets access to the morpho forms phonological encoding the communicative intention into a structure p and and it is (message is process First speech production communicative intention memory), speech production. In brief, the model ………… …

a ………… he the

early … . This lexical ecause if in a showed

GP). GP). This Interface of the speech modal modal priming: linear series of auss, auss, Chen, & speech unpacks, -

a (Kr

Kita & Özyürek (2003) turned turned into speech

holistic representation that late stage ted). and and its cognitive architecture. Kita & Özyürek (2003)

to be

information

, , (iconic) gestures (termed by

lausal structure of a language can . c 19 According According to the GP Theory, b

. symbolic stage, stage, where words are retrieved from

. . They found that, for example, ributed analytically into of it needs of the utterance (the growth point,

is expressed globally and synthetically into and and . . Therefore, according to this view, producing

es and dynamic dynamic features of concepts, which then (McNeill, 2005; McNeill & Duncan, 2000) dist representations is incorporated in the -

. . An influential theory related to this idea is t

and and phrases Lexical Retrieval hypothesis formulation tems to be retrieved from the mental lexicon (see also

of a GP spatial to be imagistic ,

(Kita & Özyürek, 2003)

the (imagistic)

symbolic symbolic part the initial “seed”

activate linguistic linguistic empirical findings - meanings conceptualization the

trajectory. trajectory. Explaining these findings, ) aid lexical access in speech production via cross - motoric of e; - imagistic part adar adar & Butterworth, 1997) language language there is no word for “swing”, the gesture will not often represent an arc The cross that the lexical possibiliti influence gesture production The GP’s idea of the unity and dialectics of spatio linguistic expressions and Hypothesis (IH) morphemes, words gesture and speech come from the same GP, they are synchronized from a of point view semantic and temporal seed includes both the gestur requires Growth Growth Point Theory According to it, the utterance grows from a constitutes lexicon and the phonological form is the genera phonological and lexicon By contrast, other models propose that gestures play a role at the stage activate the lexical i H a gesture helps speech production production process ( in a Gottesman, Gottesman, 2000; Rauscher et al., 1996) gestures Gestures According to

, , its the

sketch In by the and

(Alibali, (Alibali, , preverbal

, therefore, , where the oth symbolic esture/action conceptualizer gesture

(Kita & Özyürek,

Message Generator and and the g

(Kita, (Kita, 2000) ) , also incorporates the Information Packaging (de Ruiter, 2017) analytical thinking analytical verted into

is further elaborated in the

con motoric motoric thinking provided by (p. (p. 63) -

motor information into packages - formulator

Communication Communication Planner proposal In this model, Levelt’s conceptualizer ) are two independent streams that

20 . . Spatio This

, in synchrony with the gesture planner, . rich spatio (De Ruiter, 2000)

e the communicative intention into a

s alternative informational organization that is not , , while speech arises from formulator action action generator tive intention is generated and where the modality of organiz (Kita, (Kita, 2000) convert The to

semantic semantic structure) also generates an abstract

. (IPH) (IPH) line by linguistic formulation possibilities” that the Interface Hypothesis. - with speech synchrony. the speech generating module (

motoric thinking motoric . - Asymmetric Asymmetric Redundancy Sketch Model dations dations of the GP theory. It assumes that the Leveltian’s ginal previous version a a module suitable suitable for speaking, as Hypothesis claimed by the gesture can be “an readily accessible to analytic thinking” gesturing helps gestures gestures arise from different (cognitive) sources: gestures are generated by spatio Yeo, Hostetter, & Kita, 2017) gesture However, the process of how speech is capable to express b and imagistic information can be explained by assuming that speech and message containing the imagistic information to be gesture planner encodes the preverbal message into verbal utterance and this ensures the ori foun ( “shaped on 2003) The which formulates the exact proposition to be verbally formulated. model, generating module ( constantly interact bidirectionally so that the gestural content can be is split into two modules: (1) general the communica expression (speech vs gesture) is selected; (2) the introduced ………… …

- ………… the

of an when …

the GSA on

to overtly for speech

(Kita et al.,

at a particular ption that any

m ; ’s capability of ) cognitive processes at a given moment.

that also generates (Hostetter (Hostetter & Alibali, resistance

individuals individuals and the ld

. . For example,

gesture appearing Hypothesis and, and, specifically, with the mental mental simulati own

can be seen as a variable a

single

gesture is more likely to occur when

holds holds that 21 to say, speakers gesture because they action action generator resistance gesture gesture thresho

on the , a

representational gesture representational - s s not only speaking but

. This non Conceptualization - . That is purpose peakers have their a is based on the assumption that gestures are - affect depend for - This idea is compatible with the embodied cognition

the likelihood of a gesture :

In other words will be visuospatial or motor imagery requires activation of

. The proposal .

ing occurrent activation of the motor system (sensorimotor (sensorimotor simulation) and that gestures arise from and Gesture or -

(

information . . The GSA framework is based on the assu

by activating, manipulating, packaging and exploring spatio motor activity (Glenberg, Witt, & Metcalfe, 2013) co is action action and perceptual states as they think. Therefore, involv

th communicative context. communicative In other words, s producing a gesture threshold that the extent to which producers evoke action or perceptual state while speaking speakers rely on a stored verbal or propositional code as the basis for speaking or thinking about an idea, a gesture is less likely to occur the production speech; in engaged producing is also the system motor the height of the producer’s . This framework

• • • moment depends on motor system reflect simulate model predicts that Gestures Gestures as Simulated Action framework (GSA) 2008, 2019) thinking schematizing in general, motoric information. framework 2017) produced by a general practical actions more recent

- - of how

during during . They

relation gestures (Goldin s

ain

supporting on alignment expl for vocal tract and and integrated

and and its motor motor control boundaries are

oncurrent speech focus in fact c

account account for most

ill

. by single similarly similarly expressed in

emporal gesture they also

T above : (de Ruiter, 2017; Kita et al.,

recorded

(e.g., (e.g., GSA model)

. They w , , and ing, which also occur can be 3 and and that both speech and co (Krauss (Krauss et al., 2000; Rauscher et

Gentilucci Gentilucci & Dalla Volta, 2008 for hen constitute 2017)

(

and and itself reviewed

neural neural network connected to Broca’s

2 22 disfluent speech disfluent ; see a integration models electromagnetic articulometry

and (2) and

specifically referring to

inal inal lengthening at prosodic ová, ová, 2015 ing support support speech production re and speech cognitive cognitive processes

us will serve as further background information for the

conceptualization phase , , Tiede, & Tyrone

ć . prosodic prosodic structure theoretical manual gestures lengt

0; 0; Kita & Özyürek, 2003) : e.g., f

lexical lexical retrieval phase he t the sections

production production , or in the two Krivokapi prosodic features prosodic

(Bernardis & Gentilucci, 2006) ). ). This is in line with findings and theoretical models described so

Gesture and speech and acoustics (1) (Marstaller (Marstaller & Burian next

both modalities accompanied by and and body movements movements and a motion capture system for showed that body movements a review far, claiming that gestu process. language language and arm gestures are controlled by the same system speech gesture production engage area Behavioural Behavioural and neuroimaging studies have suggested that spoken two two studies presented in Chapters and speech with Meadow & Alibali, 2013; Hostetter & Alibali, 2019; 2014) Pouw et al., Kita et al., 2017; The al., al., 1996) 2017; Kita, 200 gestures are capable to more general internal the empirical findings on speech and gesture production, predicting that gestures serve a functional role for speaking, by speakers in the alternatively helping To conclude, ………… …

- ………… In ion

tion . This

) are …

which . Esteve locution

, peakers can (e.g., (e.g., s discourse Gibert & Prieto, /posture - speech gestures - ’s studies locutions within a locution ay also use visual

the two modalities co , and and its relation to n n other words, he

I , . However, a quest

. ) (Wagner, (Wagner, Malisz, & Kopp, align with prominent part ocution is that each level of organiza

for example, , 210

tone units : Hufnagel Hufnagel & Ren, 2018)

- 1972, 1972, 1980) , , and they m (

each , p. esture strokes, prominent parts of in turn ,

tandem

was was matched by a distinctive pattern

23 esture production Kendon with body postures

, or “paragraphs” of the Hufnagel Hufnagel & Ren, 2018) - mporal alignment . . For example, te (Kendon, (Kendon, 1980 c c structures (named

ed by a particular type of type movement ed particular a by has shown that g ” z dic structure of the spoken utterance

organized - Hufnagel, Hufnagel, 2019; Shattuck or gestural “hits” can -

Comes, Comes, Asor, Swerts, & Prieto, 2017; Esteve locution locution clusters minutes minutes video excerpt from a film made in a London pub - This suggests that g - he precise nature of this temporal (and functional)

characteri e. gestures, gestures, head nods, eyebrow movements for the same purpose. and and gesture) can work in and and

was

(Shattuck

(Swerts, Krahmer, & Avesani, 2002) ed a 2 nation seems to bear many open questions , . z s like 14) ore recent research speech 20 cues However, t coordi effect, the result of this ( employ acoustic cues (e.g., pitch accents) to mark a word, or information status to to be addressed is whether gestures and prosodic structures are jointly planned by speakers to occur simultaneously and coordinate with each other gestures gestures (peaks) of speech, and proso Gibert, Borràs 2013; Loehr, 2012; Shattuck structure structure and kinematics cluster M described how prosodi group hierarchically co (where a single speaker was talking during a group discussion with American an anthropologist). He observed that “ distinguished in the speech stream of bodily movement speech needs to be explored at the prosodic and phonological level idea was somehow present in analy prominenc

- s d the the ha

study 1990; 1990; 2007) on (

’s

between between prosodic produced

Hello” That is, meaningless

. with a peak in produced produced with they are relate -

if Hammond,

ere the gesture is 2019) ( , while duration duration and the same time, they h , speech synchrony is Pouw and colleagues - the

that that is, a downbeat to Gentilucci Gentilucci (2006) the action of expiration et al. Krahmer & Swerts

if they are to be explained . s temporally study, study,

or example, bye accompanying ‘‘ , or F -

. affect leads leads to changes in how their

produced produced speech

- affect some of the (prosodic) features . , Harrison, Bernardis & of speech

. . In that that speech and symbolic gestures (not that this gesture

(in particular on occurring occurring word. What is unclear is from al., al., 2019)

directly directly -

gesture of the target word w

waving bye (Pouw et biomechanical interdependence

operties beat showed

that gestures can have effects on the accentual cognitive cognitive mechanisms of a common underlying -

in terms of periodicity s

es , Harrison, F2 and F3 like movements with high physical impetus affect or example, - mean , F

spectral spectral pr hand hand movements y. producing producing a beat (Pouw properti

. This

that

on on the spectrum (F2) of the co mechanisms these effects originate

meaningless meaningless arm movements) influence each other when

hand hand movements do not have comparable effects. At found that gesture kinematics is reduced when co just simultaneousl effects system system that controls arms Flanagan, Ostry, & Feldman, and 1990) oral articulators on the influence that gestures and speech simultaneously produced have on each other when to to direct physical impetus that articulators goes from arm movements in to terms of oral the neural higher formants produced) strength (prominence) of the co which gesture on the showed prominence is realized in speech upward movement phase of the beat aligns peak a in envelope F0 amplitude and Also, there is evidence of a direct effect of asking speakers to produce a related muscles and this can directly of speech, like F0 and amplitude found that phonation mediated by a more direct speech and body movements repetition of Recently, it has been claimed ………… …

- t ………… are the

s wide

. The …

gesture ensures are not -

this that, while for a review of ( (Levelt, (Levelt, 1999) need to be taken into

functional relationship Corballis, Corballis, 1999; Hewes, monitoring monitoring system and and symbolic gesture -

also also includes a mechanism s

mechanisms underlying speech interaction . ,

- self the monitoring mechanism itself.

). ). A proposes, proposes, issues related to speech common common communication system and meaning. and

comprehensive model

need to be explained in a

ffs, repairs with a fresh start or a

uhm 25

o is that , - on gesture

2019)

uh ( alignment, alignment, the higher level prosody tha Postma, 2000) Postma, ed by structure view

and and mutual interactions see, among others,

( and and the self

our ion

are cod ufnagel gesture

proposals - H

y claim that spoken word - and

, e speech f what happens to gesture when speech is disfluent has gestures gestures and speech

mechanisms Shattuck related and for the speech planning processes the

refore, the

As The Gesture and disfluent speech

. abovementioned

been explored as a way to gain more production insight into the speech interrupting speech or not) and when and how to do it speech theories monitoring see The question o phoneme correction (e.g., rephrasing wrong sentences, speech segment), filled substituting pauses ( a that speakers detect the problem and deal with it speaker must decide whether to correct it in some way (e.g., by As As speech unfolds disfluencies are pervasive. Speakers hesitate in many ways by making pauses, cutting explaining governs surface phonetic variability, and between speech and gestures the and and gesture synchronizat account for the development of a accounting The mutually exclusiv integration of context. perspective that sees speech as evolving from a primitive communication system based on gestures 1973) words. words. functionally involving Broca’s area. This is in line with the language evolution meaningful/related words and that this does not happen with pseudo

- To sks . . By

. some s; s; (2) e that

lexical lexical , did not

. phase , , the studies

(e.g., (e.g., Graziano & formulation phase (de Ruiter, 2017; Kita and and in the description However

speech , , gestures should rather co Preventing speakers from phase

s s et al., 2000; Rauscher et al.,

nition). nition). conceptualization (Graham (Graham & Heywood, 1975) esture production and disfluent

. . Precisely, there is evidenc

(Kraus Graziano Graziano & Gullberg, 2018) the 26 (Seyfeddinipur 2001) & Kita, at play in the

is worse performance in linguistic tasks (e.g., result result in less efficient speech production (e.g., to conceptualization

Morsella Morsella & Krauss, 2004) , (Hoetjes et al., 2014; Hostetter et al., 2007) them from gesturing.

in fact

, of speech production . . In fact, gesture can be highly sensitive to speech shadow speech problems to be solved that is, gestures are

- yielded mixed results. For example, some detrimental whether gesture should of lexicon) or rather in

description task ( restraining they should occur more frequently during speech disfluencie , ended and resumed in systematic temporal coordination with speech retrieval contrast, contrast, two studies which elicited speech describe by motor tasks asking participants to effects of the inability to gesture on fluency have been found in a visual objects of low codability abstract lines drawings inducing inducing disfluencies) or picture naming, word recall from exploring the effects of gesture restriction defi on language performance (e.g., fluency) have previous studies have tested participants’ performance in different ta while gesturing et al., 2017; Kita, 2000; Kita & Özyürek, 2003) occur with fluent speech disentangle (as in ( retrieval retrieval phase 1996) if they are at play in the In general, the relationship between g speech has been explained in at least two mainly ways: a (1) if (lexical) gestures compensatory have role and contribute to the susp (Seyfeddinipur, Kita, & Indefrey, 2008) gesture can fore often interrupted before speech is have suggested that when speech stops, so does gesture Gullberg, 2018) disfluencies: as speech is suspended and resumed, gestures can also be Empirical studies on speech and gesture production during disfluency ………… …

. - ………… … effects Llanes

(Lucero, (Lucero, ; educational educational tigated tigated before potential theoretical theoretical models learners of English

the Also, in

and and that producing .

, their scope is not to While the cts of encouraging the use of . fluency Giménez & Prieto, 2018) - effe

not not been directly inves will be further explored in the first

(Garcia, (Garcia, Cannito, & Dagenais, 2000; 27 effects of encouraging participants to (Vilà

) test test can help their narrative abilities and have - Finlayson Finlayson et al., 2003; Jenkins et al., 2018; potential

. 2 (in terms of interword intervals, speaking time, potential provide some predictions for mixed effects were found on both fluency and have shown that asking Catalan . This issue

, retelling task rical rical studies are needed to shed light on the role of

test. The - of the thesis the of

test above speech fluency will be explored in the second study - dysarthria dysarthria

L1 L1

Aim

nted innted Chapter ).

3 1.4 prese there is some evidence that tapping or producing beat gestures can briefly reviewed of gesture restriction/encouragement on All in all, more empi gesture in the process of speech production gestures on (Chapter Coromina Coromina et al. (2018) to produce beat gestures in a training phase, improves their reading in English in the post Garcia Garcia & Dagenais, 1998; Hustad & Garcia, 2005) settings, there is evidence that training produce beat gestures children in a pre by asking them to fluency in a post with acquired total sentence duration, speaking rate, and phrasing strategy by triggering a more natural speech chunking Yet, Yet, facilitate lexical retrieval and Zaharchuk, word & recall Casasanto, from 2014; definition Ravizza, gestures in 2003) speech rehabilitation can enhance intelligibility of patients study study On the other hand, the gestures on speech and fluency using story retellings speech content and planning ( Rauscher et al., 1996) find effects of restraining gestures on fluency or speech length. In studies

es and and two and

s gesture on features

a role in

s qualitative approach reported in

encouraging acoustic speech prosodic some and and will report on

the effects of previous studies

other prosodic , the results speech speech

5 . report of the speech production methodological

and restraining ture ture production play

will spontaneous spontaneous speech production the

- investigate

f spontaneous narrative speech. o how to -

s i 28 semi For example, in the speech production

Chapter 4 the effects on . understanding

further evidence on the possible effects of The next two chapters

in clarifying this issue clarifying Finally, Finally, in Chapter carried out on the data collected for the two

some aspects impact on fluency dissertation dissertation

can In addition, formulation formulation of prosodic patterns and in the final Therefore, aimed at exploring .

will be discussed in light of ) 3 d contribute to to d contribute prosodic prosodic patterns investigations. will discuss and and

. 2 of the present

woul

help provide a better planning and goal

this dissertation hypotheses (Chapters and quantitative analys studies and used in the process process in his full unfolding. twin investigations the use of gesture The restriction/encouragement Crucially, evidence of these effects in semi would articulation phase. gesture restriction and encouragement on fluency and features underlying process, it is not clear whether and how ges the make direct predictions for ………… …

…………

paper: … peech S ).the Exploring estures ffects of ffects G E on underreview se of U

29 the This chapter is based on the following the on based is chapter This

Exploring the the Exploring

Effects of Restraining the Use of Gestures on Speech on Gestures of Use the Restraining of Effects estraining Cravotta, A., Prieto, P., & M.G.,( A.,P.,& Prieto, Busà, Cravotta, R

Abstract of gesture examined: and and shows

gesture. gesture. Further results

yielded mixed results. To corrections, repetitions, - has that also takes into account a

t t al.’s (2014) ike pitch or intensity, but did not

30 . . Moreover, previous studies have used Hoetjes e

eech. Twenty native Italian speakers described the . However, this research ence was found that the speakers’ inability to gesture

This result expands spontaneous spontaneous narrative speech in terms of discourse length,

(Hoetjes et al., 2014) - Research in gesture studies has investigated whether the on more naturalistic speech tasks The present study investigates the effects of restraining hand No evid

semi

s lusion: speech does not become longer, more disfluent or monotonous when c work is needed to shed more light on the direct influence of gesture on speech. Con that participants cannot gesture compared to when they can Results: affect features. acoustic and fluency seconds), seconds), disfluencies (filled insertions, interruptions) and prosodic properties related pauses, to speech rate, self intensity. F0 and Restraining Restraining gestures (N); Restraining gestures (R) (i.e., the speakers had to sit on their hands). The following correlates of speech were speech discourse length (number of words and discourse length in Method: gestures on narrative sp content of short comic strips to a listener in two conditions: Non very very controlled tasks and evidence is lacking about more speech. Thus, spontaneous further research is needed to assess the effects restriction intensity. F0 and including measures, prosodic of complete set acoustic our our knowledge, only one study analyzed whether restraining the use of gestures impacts on acoustic measures l find effects Purpose: inability to gesture is detrimental to speech, as suggested by theoretical accounts the main ………… … , - - - ………… have

. Also, … Meadow, Meadow, - Hostetter & in express the ,

language language and , speech gesture - directed role of - conceptualization - manual gestures for - and and co re

iter, 2017) , as Specifically, Specifically, gestures and

. parts of speech, e.g. pitch nts Hufnagel Hufnagel & Ren, 2018) Gibert et al., 2017; Esteve

- - (Cook, (Cook, Yip, & Gold (de Ru Interface Model, Kita and Özyürek,

(Esteve

31 motoric motoric level (Krauss, (Krauss, Chen, & Gottesman, 2000) rmation prosodic prosodic level

In fact, there is evidence that .

Gestures Gestures as simulated action framework (

lengthen during speech prominence and at boundaries et al., 2017) . . These effects are still under investigation from different and and reduce cognitive load

ć among many others, , Meadow, Meadow, Nusbaum, Kelly, & Wagner, 2001; Ping & Goldin

- Introduction Kita, Kita, Alibali, & Chu, 2017;

gesture gesture production have proposed that gestures contribute to 2.1 - , facilitate lexical access Krivokapi th speech production at the e.g., production production engages brain areas that are functionally connected to Broca’s prosodic prosodic boundaries extends to body moveme been shown to ( action are closely related on a recent recordings of concurrent speech electromagnetic and articulometry for body vocal movements tract movements (using and a motion capture system for body movements) have shown that final lengthening at example, example, gestural strokes or prominent parts of gestures (or gesture ‘hits’) tend to align with prosodically prominent accents (e.g., Gibert & Prieto, 2013; Loehr, 2012; Shattuck Gesture Gesture production has also been shown to be strongly wi interconnected prosodic units are synchronized from a temporal point of view. For 2012; 2012; Goldin Meadow, 2010) perspectives disciplines. and speakers’ mental simulation of motor actions and perceptual states during speech production Alibali, 2019) utterance utterance planning and conceptualization (Gestu hypothesis, 2003) provide additional spatial info Research in the last decades has investigated the gestures self in the process of speech speech production. Theoretical models for

; . its not

or any

acoustic have inhibiting

) provide specific , the effects on ensity between between gestures

agery agery content) and, one of the methods effects of often reported in the (i.e., (i.e., head, face, hands)

F0 and int , as well as speech content interaction theoretical theoretical models mentioned

Gentilucci Gentilucci & Dalla Volta, 2008

]. (e.g., (e.g., see

; though though it does not that restraining the use of gestures 32 This study is , 2014) (Hoetjes et al., 2014)

( .

2015 (e.g., (e.g., participants, procedure, etc.), , some of the

of speech á 2014)

participants participants were asked to speak while trying to . . In the present study, we aim to examine the intensity.

The features Burianov

Wagner et al. Wagner et al. .

to to explore the interrelation between gesture and speech and F0 and Krauss Krauss et al., 2000; McClave, 1998; Rauscher, Krauss, & the expressiveness and richness of speech, as well as

Dobrogaev Dobrogaev (1929) and s

acoustic ). ). [For a general overview on the . However, in general, how the inability to gesture impacts 1.3 was Previous studies on restraining gestures Marstaller Marstaller ly affect to to test the predictions of spontaneous spontaneous speech is still unclear, and the ( -

quantitative analysis avoid all possible body movements while talking literature literature (e.g., Chen, 1996; Wagner et al., details about the methodology acoustic acoustic features of speech such as F0 and intensity has been overlooked One of the first studies to claim direct fluency, assessed assessed in relation to fluency, speech length (i.e., semantic richness, spatial relations expression, im with the exception of one study features such as as features such The potential effects of restraining gestures’ use on speech have been received much attention impact of gesture restriction on narrative speech, with respect to fluency and speech length. The study also explores how this can impact in Section production semi gestures on through through gesture restriction during speech production, for example, sitting on their hands or by folding their arms) has been used for a review speech, seeand also Investigating how people speak when they cannot use their hands (e.g., area ………… …

, ………… In

By By .

1 … As for lexical subject subject - thus thus used

Table , , studies have in participants were results results are mixed.

speech production sfluent speech. i g gestures on speech fluency more recent empirical by directly testing summarized were still showing rhythmic

recall recall from definitions. , in which

or s s (with both high and low verbal ,

, they

33 successful fluent below and

investigated investigated the effects of restraining hands After this study,

Dobrogaev

2 s of restraining gestures on of task and different gesture inhibition methods). described s e prevented from gesturing, in the description of absent absent objects, produced more d and and key component of Graham Graham and Heywood (1975) focused focused description tasks) body body movements (a) speech lost expressiveness and richness;

is only available in Russian. in available only is different types

, a

, , the studies were aimed to test different hypotheses and We thank Mariia Pronina for providing a detailed summary of Dobrogaev (1929) that, to our

knowledge,

participants who wer both visible contrast, in asked to describe abstract lines drawing the studies that examined fluency in connected speech, Morsella and Krauss (2004) use during an object description task. The study showed that the using very retrieval (Hagoort & Indefrey, 2014; Indefrey, 2011; Kearney & Guenther, 2019) using tasks such as picture naming or word This makes it hard to compare the results and to draw a generalization. generalization. draw a to This and makes compare the results it hard to With regard to the effect addressed the issue by either focusing on connected speech (typically general heterogeneous designs and methodology (e.g., between or within designs; and and (b) speakers had difficulties with word retrieval resulting in short and disconnected sentence investigations have assessed the effects of restrainin production. These are gestures gestures and movements in different body parts (i.e., fingers, eyes, head). The main findings were the following: when speakers were asked to try to exclude all however, as observed by 2

- n by

In a Kita

while lexical lexical

, children and

. a a package), Rauscher et

. that ions when they ping that when gestures found found that speakers

in terms of pauses, found found

the task while sitting

wrap

This was confirmed ing found Hostetter, Alibali, , 2007) (

increased the relative frequency juncture juncture filled pauses than the

- Kirk

hand hand gesture restriction on speakers from gesturing during oral 34 and proportion of total speech time spent

the effects of hands’ immobilization on

speakers had to give instructions on how More recently,

ing . The study did not find effects of the inability d). d). rohibited from gesturing during the descriptions. explor Pine, Bird

Horbury and Guttentag (1998) , - , in which however, the the effects of

; other factors such as mutual visibility and previous ( Frick

2014) , ( there was no difference between the restraining and no found found that preventing ,

naming task juncture juncture filled pauses in speech with spatial content - analyzed analyzed the spoken productions of participants that were asked to -

more likely to generate target words from definit ing was significantly higher in the restraining condition 2007)

were free to gesture than when they were prevented from gesturing. picture With reference to retrieval tasks were to tie a tie, with half of the participants perform on their hands experience were also teste (in terms of speech filled fluency on and gesture rate pauses) to pauses or a higher percentage of non participants whose hands were not restrained. Hoetjes et al., ( describe how to complete three motor tasks (e.g., with half of them being p The participants whose hands were restrained did not produce more filled a a similar task (i.e., animated cartoon retelling) were restrained, speech was overall more disfluent repetitions and reformulations al. al. (1996) descriptions of animated action cartoons of non Finlayson et al. (2003) restraining restraining conditions on any of the measures of fluency (e.g., hesitations, filled pauses, etc.) paus codability) ………… …

. - e ………… in by the , - f . word word … 1984) o ( resolv - Church, Church, restricted to Beattie and

spontaneous spontaneous

when when they were arms group arms

- s found found that gesture Rimé et al. vs

based text analysis

. - ants had than those who were math explanations minute by , , testing the idea that

- 8 geometry conjectures

TOTs in the fold

. . A study whether tivity/movement tivity/movement (both measures with their arms folded had than participants that were free to

(Brown, 1991, p. 204) 1991, (Brown, gesture particip 35 gestures gestures in 50 - speech speech content were free to gesture ranscriptions ranscriptions of the speech produced to based computer program). By contrast,

- - participants and and when they were free to move.

The t

and and explain .

participants than did not experience more TOT s : when they

3 Nathan, Nathan, Chelule, & Wang (2019) TOT he results showed that the participants who were ) and and why found found that restraining gesture use does not affect

compared to , , and that free children led to a lowered imagery level in the words used, as well

speech speech length (TOTs

Precisely, t a more fluent retrieval process states

from definition

However, participants participants were analyzed through dictionary inability to gesture can in principle affect the speaker’s selection of ongue Type of problematic lexical accessing event, experienced as ‘being sure that the information is in

memory but (…) temporarily unable to access it’

restriction restriction does not impact Participants had language to assess use in were true or false the as as a reduction of content related to ac were obtained via a dictionary Walkington, Woods, information and the words and structure used Alibali, & Kelly, to 2017; Kita convey et al., 2017) it ( found that restraining speakers’ conversations A A number of studies have investigated the potential effects of the inability to gesture on the prevented from gesturing experienced fewer free to gesture more proportionally Coughlan Coughlan (1999) recall general gesture. T not. from gesturing, named more words correctly and resolved successfully more Tip 3

- ; ; LIWC, Graham explored restricted

- to test the

(e.g., (e.g., filling motor motor task

Hoetjes et al. , , elicited via a Finlayson Finlayson et al. Jenkins, Coppola n increase in the and and

. . With regard to

esults esults showed that striction in terms of found found that speakers, if Graesser, Graesser, (2013)

and and Casey (2001) ). ). The r 07) Similarly, Similarly, a

. story story retelling task narrative speech a ere restrained during

Emmorey kind and number of words used and 36

gesture gesture speakers produced more verbal - Hostetter Hostetter et al. (2007) ed to - free condition - study study using

abstract lines descriptions includ

Hostetter Hostetter et al. (20 in In a

rovide evidence that free trials in any of 148 different language measures - p McNamara, McNamara, Louwerse, Cai, &

motor motor events (e.g., putting one lace over the other vs. - giving commands to solve a spatial problem . However, metrix, to to object orientation, while speakers that were prevented from - ontent ontent phrases in the speech produced in the gesture . These measures reported a higher number of words used and higher number of did not find differences in the amount of speech produced (i.e.,

motor motor terms used was found. - io e.g., e.g., number of novel propositions, episode length) when gestures are a a puzzle grid with blocks), free references spat whether gesture restriction affects spatial and found that motor in expressions and unable unable to gesture, produced less detailed (semantically rich) speech when describing spatio crossing the laces over one another); but no difference in the number of narrative) narrative) and organization. Both (2014) number of words) when gestures w descriptions ( restrained, but it is negatively affected by gesture re grammatical complexity (i.e., number of subordinated clauses in each and and Heywood (1975) speech length as well as speech structure and content, and Coelho (2018) story retelling task, does not change in terms of speech length or content spatial c condition than in the hands number of words used for expressing spatial content was found in assessed other speech patterns. effects of gesture inhibition on the speech (2003) produced, Pennebaker, Booth, Boyd, & Francis, 2015) there was no difference between the speech produced inhibited vs gesture in the gesture tools (Coh ………… …

: ………… : : WS R

: WS Add :

WS : 10 (f, (f, 10 :

Add … sitting S : 38 pairs pairs 38 : (young) + + (young) Add: S R:

(WS) : Dutch Dutch :

: English English : L

: Sitting on on Sitting : : 20 : : 108 (f, m) m) (f, 108 : L Condition: : English English : S S R L R Condition R English. R Condition subjects L: R condition: - : Gripping bottom of of bottom Gripping : : participant. participant. : R Description of routes on : true/false statements on 8 8 on statements true/false : (pictures story Retell : task motor of Description : Add + givers instruction (f, m): addressees. hands. T conjecturesgeometry + justifications; interviewer Hands in oven mitts attached table; a to Within T sequence) Unfamiliar listener. m) seat. the T: a map. participants 19 (elderly), f, m. on hands. T tie). (i.e., a tie Design, task ining gestures on speech speech on gestures ining

speech speech

descriptions descriptions of routes on a nformation nformation in both conditions.

37 Elderly speakers produced more spatial incontent R. No effects on number of words N of words per sentence, word concreteness. Speech less grammatically complex organized.effectsand worse No on speech length and content. Content; Speech length No effects on language use in math 148 on Based justifications. measures (from text analysis software), including found that, in the found that,

- No effects on fluency and F0 (nor intensity) - Fluency; Lexical retrieval performance -

provides a summary of the studies reviewed above, giving Özer et al. (2017) Özer et

. groups groups of young vs elderly adults speakers; nonetheless, elderly 1 Summary of Summary previous findings on the effects of restra

1 (2014) (2014)

, , gesture restriction did not impact the duration of routes descriptions

s ksun (2017) zer,Tansan, zer,Malykhina, ö

Hoetjes, Krahmer & Swert Ö Ö Chatterjee & G Jenkins, Jenkins, Coppola & Coelho (2018) Walkington, Woods, Nathan, Chelule, Wang (2019) Study Table theirdetailed information about research design. Table older). to recent more (from adults adults produced more spatial information (i.e., street name, landmark or direction) when sitting on their hands than when free to gesture, whereas young adults expressed comparable spatial i rotation map in two gesturing were more likely to also lexically specify the direction of the :

S L: R

R ish ish L: BS R:

: a R WS sitting R -

: Engl :

English. L Add:

R: Add:

Add: : English. English. :

L R Velcro cotton . L:

Experimenter; confederate : WS :

Folded Folded arms. WS BS Between R:

: BS : English Armchair with English 79 (f, m) R condition: R: (BS)

L: L: R: S: 36 (f, m) (EXP1) + Add: Add: S:

R condition: - R condition:

Description of animated Give command to solve a Retrieval of lexical items Description of motor Description of visual Retelling a cartoon Holding a rod (EXP1) and Dummy electrodes on 30 (15 f, 15m); 26 (f, m) 6 (f)

: Retrieval of lexical items items lexical of Retrieval : : Picture naming naming Picture : : : 65 children (f, m). m). (f, children 65 : T: L:cartoon English confederate. S: 41 (f, m) Dummy electrodes on hands. velcro strips arms.on condition: T: spatial(filling problem puzzle grid with blocks); English S: on hands; T definition. from 60 (f, m) condition T: definition from Add: 18 (f, m) (EXP2) R: wearing an apron with Velcro (EXP2). condition: T S Gloves with Velcro on table. R condition T: package) a (e.g., wrap task English S: board. gloves wooden on condition: subjects T objects Offline R: arms. T: (video) Participants S:

lexically lexically phrases

- - specify rotation direction - More spatial content Longer speech (number of words) Speakerslikely more to Less semantically rich verbs. No effects on number of spatial motor terms. No effect on amount of speech - -

better

and number of and number TOTs). resolved Different Restraining methods of EXP1 and 2 have similar results. Spatial content more (only) disfluent - TOTs* less experienced = performance Fewer lexical retrieved items on effect (no number of TOTs Speech more Speech more disfluent No effects on filled (e.g., fluency pauses) Speech more disfluent less TOTs* TOTs* less resolved =worst performance

Horbury - auscher, R Krauss & Chen (1996) (1998) Frick & Guttentag Beattie & Coughlan (1999) Emmorey & Emmorey Casey, (2001) Finlayson, Forrest, Lickley & Beck (2003) Morsella & Krauss (2004) (2007) (2007) Pine, Bird & Kirk (2007) Hostetter, Alibali, & Kita ………… … .

…………

he R: t

16 that

S: …

f: WS: English ;

English.

is, to our

L: claim L: R condition:

tion of abstract abstract of tion coustics ell, by hearing Armchair devised to R Condition: a minutes spontaneous Experimenter. Audience S: 6 (m). -

R: 2019)

50 (

T: conversation Add: (m) (head, restrain movements limbs). T: Descrip drawings. lines Add: Folded arms. WS mentioned) (not number number of subjects

of speech like F0 and

S: peech s whether speech becomes

was gesturing or not while Restraining Gesture condition; on

Hoetjes et al. (2014)

Addressee;

. Add:

ntion

R condition: 39

acoustic features

roduction Language;

Reduction of vocabulary size (i.e., richness). More spatial content words and phrases Less use of demonstratives so’) ‘like this’, ‘like (‘there’, Decrease in imagery content and movement/ action content. , Paxton, Harrison & Dixon Task;

esture there were no effects of restraining gestures on the bjects. This review excludes studies on restraining gestures where the g

one study that investigated Pouw

codability

that - Restraining gestures’ method;

not not received much atte between between su

Tongue states; R: - disfluent (i.e., short, disconnected sentences); of reduction expressiveness. Increased total total Increased time spent speech pausing low In items descriptions=more hesitations, more pauses, more words. Speech more Marginal reduction of speech rate BS: the -

have of

- males;

Influence of of Influence Tip m:

TOTs: the speakers’ voice only, whether someone speaking. However, speakers’ pitch range nor on any other acoustic measure (i.e., max, min and mean pitch and mean intensity). Also, the speech data perceptually was and showed tested that listeners were not able to t intensity knowledge, the only more monotonous (in terms of pitch range) when speakers cannot gesture The study found While a number of studies have investigated the potential effects of the inability to gesture on fluency, speech length and speech effects of restraining gestures content, on speech produced was not directly was analyzed. produced speech (1929) (1929) * females; within subjects; Dobrogaev Graham & Heywood (1975) Rimé, Schiaratura, Ghysselinckx & Hupet (1984)

. voice motor motor

affect and and spectral

showed that

- tilucci tilucci (2006) s

Gen the evidence that

showed that speakers’ occurring speech. That - and

de envelope and a peak ) acoustic realization of movement reproduce speakers’ arm Gentilucci Gentilucci & Dalla Volta 9 . . The study showed that

listening to the recordings to listening rm see (201 Swerts (2007) related to longer longer durations ;

- is Bernardis biomechanical interdependence , tight relationship between gesture just by just and

et al.

listeners to

40 predicted from their speech recordings.

produced produced with meaningful/semantically - for a proposed that visual beats have a similar

manner to changes in the

to phonate while performing movements of

hey d Krahmer

. T Burianová Burianová (2015) For example, ggested ggested by both behavioral and neuroimaging and and this could directly affect prosodic metrics of

exploring the

) Also, Pouw, Harrison and word led

as su

were asked , synchronize (in terms of vowel duration when words were co

lower F2, F3)

- of speakers producing steady vowels increase). y y is based on the finding that a Marstaller Marstaller for a review

(

) uch effects on the acoustics allow 2008 speech. between gestures and the acoustic realization of co is, hand gesture movements could affect involved in the expiration, actions of the muscles merely moving arms affected the acoustics of moments in phonation time (i.e., a downbeat to upward movement phase of the beat at particular gesture temporally aligned with a peak in amplitu in F0). This provides evidence emphasizing function as pitch accents. Furthermore, in an where speakers experiment different strengths, producing producing a visual beat (head nod, eyebrow movement or hand gesture) on a given target prominence properties showed that related gestures, F0 and spectral properties of vowels were enhanced (F0 and F2 spoken language and arm gestures are controlled by control the system same studies ( movements in a ina movements This line of research, production and prosodic modulation The stud acoustics s arm movements of speakers can be ………… … ………… the the low

been

trong

… ve s s and found found on

ha investigate

been eech in more am am & Heywood, escription of visual on fluency spontaneous narrative

- investigated investigated the effects (Grah Finlayson Finlayson et al., 2003; outcomes. outcomes. For example, (

effects have tasks may interact with have

in semi

; spontaneous sp - functioning. functioning. The study focuses on mixed ity to gesture might be les , such as the d

studies

Morsella Morsella & Krauss, 2004)

kinds of (Hoetjes et al., 2014; Hostetter et al., tasks can be useful to investigate 41

yield different its full

d and and is particularly aimed to test whether an a more ecologically valid setting. valid morea ecologically previous , different 1

to elicit semi story retellings

two two studies which elicited speech by asking esearch esearch question r

. The Table impact of the inabil

and and controlled tasks in features of speech specific e tasks are purposefully designed to investigate speech shown

visual visual objects description ( es the , , the description of low codability abstract drawings or routes on a in did not find effects of restraining gestures on fluency nor speech Summary & Summary . . In studies using

. . By contrast, acoustic sum, as ome detrimental effects of the inability to gesture the the inability to gesture affects speech production with respect to fluency, Our Our study builds upon previous research investigating the effects of the inability to gesture by eliciting narrative speech as a way to the speech production process in speech and less evident. Nonetheless, such in process speech production 2007) length both fluency and speech content and planning Jenkins et al., 2018; Rauscher et al., 1996) codability codability abstract lines drawings descriptions 1975) participants to describe motor tasks production production processes speakers’ inability to gesture s found cartoon cartoon are designed ecologically valid settings and planning and involve production mechanisms without challenging specific speech more comprehensive speech objects map. Th production in relation to, for example, spatial memory or lexical retrieval. Other tasks, such as story retelling or the description of an animated In of restraining gestures on speech using different kinds of tasks. Some of these are focused

- - Non in which

fundamental fundamental ence of individual condition condition (R)

speakers were free to gesture speakers were from the same university. As in which participants had to

Meadow, Meadow, Newcombe, & Shipley, -

. Briton Briton & Hall, 1995; Chu, Meyer, which, to our knowledge, has only ( in which

former student prosody narration task speech discourse length (in terms of number of subject subject design (with a within subject factor:

- production

Restraining gesture

and and Hoetjes et al. (2014) Hoetjes et al.

. ; Method

rrating Participants 2.2 dis, dis, Nagpal, Marentette, & Hauer, 2018; O’Carroll, Nicoladis, & a a and 1 of them was v o fulfillment of course credits or a free breakfast. Only female participants They were all from the Veneto Nineteen region (age of M them = were 24.1; undergraduate SD students Pad = 5.5). at the University of compensation for their participation they were either given partial frequency and intensity. and frequency Twenty female native speakers of Italian participated in the experiment. words and story length), fluency (in terms of number of filled pauses, self corrections, repetitions, insertions, interruptions, speech rate) silent as well as pauses, a set of and speech features related to Nicola 2015) Smithson, The analysis will focus on: differences in gesture Foulkes, & Kita, 2014; Goksun, Goldin 2013; Hostetter & Hopkins, 2002; Hostetter & Potthoff, 2012; Kita, 2009; when na participants were asked to sit on their hands while telling the story. The experiment has a within Condition) in order to control for the unavoidable pres The present study used a watch and describe a set of comic strips in two different conditions: restraining gesture condition (N) been explored by speech length and speech ………… … - - ………… … comic , , and the Simon’s Cat by

. s, s, namely (a) to control for an example

2 Figure

43 The comic strips were carefully selected ). ). task task (see as as recorded with a HD video camera (JVC GZ scene narration). Moreover, Simon’s Cat - item effects, the target comic strips were shown in scene comic strip for the experiment (from scene comic used -

Simon Simon Tofield, reproduced with permission).

). f presentations which were counterbalanced across conditions (Briton & Hall, 1995; Hostetter & Hostetter Hopkins, 1995; 2002) &(Briton Hall, related differences in F0 values; and (b) to control for potential ection for the complete set - scene comic strips adapted from Simon’s Cat by Simon Tofield s

Example of a 4

Procedure Materials 2 next Figure HD7E Everio) and speech was recorded sampling rate) (16 as a bit separate audio .wav track using files, a MIPRO 44.1kHz wireless head The participants were tested individually in a quiet room at the University of Padova. Each session w To control for potential two orders o (see strips do not contain text but feature a variety of characters and represent many motion events. Presumably, this property of the comic strips would make participants describe the events and spatial relations using gestures. and and adapted so that they complexity and length (4 were considered equivalent in terms of were used for the narration Appendix than males males than Sixteen 4 for gender gender differences in gesture production, as it might females be are the more case expressive and that produce more gestures when speaking were recruited in the study for two main reason

. s sat at

sequence - participant

instruction he t

, 3

Figure

listener as if he was also a

the right). shown in

As 44 ross ross data collection. A second video pack pack transmitter connected to a Zoom - be covered up. Then you will have to room and was first given an informed

“You “You will be shown a set of short front of the speaker ( speaker the of front with a body

audio audio mixer. All levels were set prior to the first participant up: the image theshows speaker (left) and the addressee iswho sitting in

- Set

3 stances were kept consistent ac Figure describe the story in sufficient detail so that your partner (who does not know the story) is able to reconstruct it by placing four comic cards that comic comic strips. A cat and its friends are the protagonists. Take your time to look at each of the short strips. When you think you understand the story they depict, the comic strip will consent consent form to sign. She was introduced to fellow participant. Both the participant and written instructions. The participant the received the following listener were given (translated from Italian): Each Each participant entered the Di camera was placed in front of the listener upper and body recorded and face the during listener’s the whole session. The experimenter sidethe for participant’s the entire experiment. recording her upper body and face. sat on an office armchair and interacted with a listener (a research confederate assistant) that sat in front of her at a distance of 1.50 m R16 R16 digital collection. data throughout remainedand consistent The camera was set in front of the participant (at 2.50 m distance) mounted microphone ………… … ………… he … cted cted to provide : : (1) the participant The reason why we made

as well as to give ecological the correct order based on the

. . By contrast, to ensure that the )

face interaction; (3) the listener was 45 step step sequence - - to - 2007) m confident with the camera. Specifically, three trials trials to show the participant how the task should

hown that gestures can be adapted depending on the scene comic strip to learn the story it depicted (for (Holler & Wilkin, 2009) - narration task. In this way, the participants felt an

(Jacobs (Jacobs & Garnham, familiarization mic strip was then concealed and the subject told the story to the

cipants cipants believe the confederate addressee was a fellow participant reconstruct it by putting the four images in speaker’s story. the co confederate addressee in a face then given four cards, each showing one scene of the comic, and had to be performed and to make the each trial consisted of a examined a four approximately a minimum of 5 seconds to a maximum of 40 seconds); (2) Each Each participant had to retell a total of 16 written stories. instructions were To clear, make the sure experiment the started with a set initial of 2 allowed allowed to interact more with the participant after the narration task and while solving his part of the task, i.e., when he was reconstructing story. t fact, it has been s addressee’s feedback (e.g., lower gesture rate when addressees are less attentive interaction between participant and confederate was natural, he was basic basic backchannel and feedback cues to the speaker while listening to the stories (e.g., nodding when he felt it was natural to do so, while avoiding asking for clarifications and showing either amusement or boredom). In validity validity to the obligation to explain the story clearly and fully participant” was dependent on them to understand it in order to finish the because their “fellow comprehension task. The confederate listener was instru parti who did not know the stories in advance was to avoid potential effects of common ground make up the strip in the correct order”. rather

On On the sitting on

person speakers were free up up cannot exclude - Restraining gesture e., (i.

), ), and the second half (2

task, more comfortable with strips were carefully selected and

was was kept the same (N, R) for all 46 N condition

s ). ). Asking speakers to sit on their hands (as ition

. 2001; 2001; Hoetjes et al., 2014; Özer et al., 2017)

restraining gesture condition R condition -

es the argument that the N condition was not trained story story familiarization phase, the participants had to tell the - Non (i.e., (i.e., participants are asked to sit on their hands while while narrating; hence,

s. s. Even though the comic Emmorey & Casey, adapted adapted so that they were equivalent in terms of complexity and length, in order to avoid potential item effects half of the participants explained In the R condition participants were asked to sit on narrating via written their instructions and an illustration of a hands while her hand familiarization trials plus other two familiarization condition trials exclud before each be to withenough comparable the condition. R condition condition could be more familiar with the the setting/the listener than in the N condition, with possible effects on their productions. However, the presence of two initial general natural natural and it would lead to carryover effects between R and N. other hand, we are aware that this experimental set possible order effects due to the fact produced that after the the R condition N is condition. always For example, participants in the R The order of the two cond participants: this is because we believed that telling participants to “come back” to a N condition after having restrained their gestures’ use was not than simply asking them not to gesture, was meant to limit the imposing additional risk cognitive load resulting of from the need of consciously remembering gesture to not extra extra familiarization stories + 5 target trials) in the condition narrating; hence, in first half of the comic strips set (2 extra familiarization stories + 5 target trials) in the to gesture After the two ………… …

………… es 10 and and

… × can be functional paus - Zellner (1994, p. 48) this study, we used a le the same comic strips

define fluency as “the ability to talk

corrections.” Thus, fluency

s s disfluency. In - 47 2014) (

ginning ginning and at the end). A measure of audio ere counted. . . In addition, according to th w

Wang

(p.93)

and

per story

disfluency measures

Data Analysis: Transcriptions, fluency and acoustics wordtokens length length with few pauses, (…) to fill time with talk. A person who is of a set of features that characterize measure of speech rate which was automatically obtained using a Praat and and make speech errors and measured self not only by measures of speech rate (that gives a general idea of the efficiency of the speech production process) but also by the absence at fluent (…) does not have to stop many times to think of what to say next or how to phrase it” “people are disfluent if they often hesitate, make non Fluency and Fillmore, Kempler are excluded both at the be file duration in seconds was included as a measure of speech discourse length (or story duration). The contents were manually transcribed the The recordings were edited so that a separate short audio file was created for each story told. Each audio file starts at the moment the participant starts telling the story until the moment the utterance ends (i.e., silences condition and 39.8 in the 39.8 condition). R and condition Speech discourse leng The experiment lasted approximately 30 minutes. Audiovisual recordings of a total of 200 short narratives were obtained (20 participants target trials) lasting a total of 77.5 minutes (37.7 minutes in the N were explained in the R condition by the other half of the participants. By this we made sure that comic strip materials were counterbalanced across conditions. half of the comic strips in the N condition, whi

s “il

per

”; . s “nella “Some

;

, ” entrambe . (examples

i -

sulla fronte n t

r of syllables tu

a lei per o …

a a lot of birds in front of her to

uccelli”, “there is a

–

acquarium acquarium again”), di uccelli Moreover, based on

… stanno stanno mangiando delle

di di the ni - per terra”, “he takes off his h

t bel po’ both portions of food” a single annotator

- on theon ground”);

of any of the following types of al un

; by

) –

… 48 e lo butta

. . Specifically, the script detects potential a… the man” , instances interruptions interruptions of a word, or pronunciation of ) and throws itand (e.g., (e.g., “degli uccelli

peaks in intensity (dB) that are preceded and

…

in un prato ;

e lo butt nually nually annotated “the fish is inside : abrupt birds”); and repetitions of longer segments (e.g., “si , : : of sounds (e.g., stuttering; “il pesce è di nuovo dentro

(Bergmann, (Bergmann, Sprenger, & Schmid, 2015; Götz, 2013; : : of words or phrases when speech needs further of eating eating bread crumbs on the ground

in terms of the forehead of

, among others of and throws itand nascosto nascosto sopra l’armadio (…) graffia l’uom elo”,“in the next scene it holds the mouse …

n acquario” l’ ollar - to give it to her”) give it to to le porzioni di cibo” , “the cat eats scena successiva tiene sollevato il topo davanti dargli toglie il collare c Insertions qualification or detail briciole di pane birds are gatto l’uomo” “The cat hidden on top of the wardrobe (…) scratches the ma Interruptions an isolated incoherent sound (e.g., “il gatto mangia l repetitions of words (e.g., “c’è un gruppo group Repetitions (De Jong & Wempe, 2009)

• • •

from our data are reported below in Italian and translated in English for of theconvenience reader): speech rate is given as number previous of studies syllables/s). Kormos, 2014 disfluencies were ma syllable syllable nuclei followed by dips in intensity. It then divides produced in the each numbe audio file by the file’s total duration in seconds (i.e., script ………… … ………… ncy … F0 floor software based (a (De Jong -

Praat with a mean of

based (slip of the - cat”);

e stribution stribution curves suggested , th , en ee

th . . Thus, we decided to set

49 based (e.g., rephrasing); lexicon - il gatto”, “ il gatto”, ia a publicly available Praat script by Jonas

. . To explore whether fundamental freque pronunciations);

. : syntax (sounds (sounds like “ehm”, “mmm”) and prolongations of annotated annotated automatically by a Praat script (as (as reported in the literature review provided in

: alloraaa

(e.g., number of filled per(e.g., pauses 100 words). corrections - word is replaced with another word); phonology or unclear tongue Filled pauses “ (e.g., vowels Silent pauses: & Wempe, 2009) Self

• • • measures. measures. After setting F0 floor and ceiling, the F0 metrics were extracted for every audio file (story) v approximately approximately 50Hz Blomgren, Chen, Ng, & Gilbert, 1998) and ceiling to 100Hz and 500Hz respectively for all participants. Setting the floor to 100Hz allowed us to avoid vocal fry effects on the F0 Previous literature has shown that for female speakers vocal fry register excursions fall in a low frequency F0 range that is generally below 100Hz (Hollien & Michel, 1968; McGlone, 1967; Murry, 1971) Praat Praat for every audio file. The F0 data distributions examined for each were speaker individually; the di plotted and that overall, modal voice register was centered between 100 and 500Hz. The acoustic analysis of speech was (Boersma & Weenink, done 2018) using the (F0) and intensity were modulated conditions, differently a set across of pitch the and N intensity measures and were R extracted with The absolute count of all types of relative measure disfluencies was converted into a Acoustic analysis

It then

. in descriptive (N, R).

the ma

correction correction rate, (7) Filled - h Variation Quotient (PVQ), shows

2 maximum intensity, as well as

Table 50

. . PVQ is a metric derived from the F0 standard deviation. deviation. (5) Interruption rate, (6) Self

(Hincks, 2005) he script extracts a pitch value every 10 ms of speech via T

Statistical analysis same way, intensity listings were extracted with an adapted version . 4 https://github.com/YoeriNijs/PraatPitch

(16) Minimum intensity, (17) Maximum intensity, (18) Mean intensity, (19) Intensity standard of each of the by Condition separated 19 variables statistics pauses pauses rate, (8) Silent Pauses rate, (9) Total Disfluencies rate (including 3, 4, 5, 6, 7), (10) Speech rate, (11) Minimum F0, (12) Maximum F0, (13) Mean F0, (14) F0 standard deviation, (15) Pitc The data analysis focused on a total of 19 variables of interest: (1) Story duration (in seconds), (2) Number of words per story, (3) Repetition rate, (4) Insertion rate, file and, subsequently, mean, minimum, were deviation computed. standard In the of the Praat script mentioned above which works similarly to the one used for extracting F0 metrics: loudness listings were extracted for every audio measure of pitch variability, Pitch Variation Quotient (PVQ) computed was also deviation, which is expressed as a percentage of the mean (see who2005, measure proposed of perceived this a metric as liveliness). Hincks autocorrelation autocorrelation algorithm for the whole audio file (story told) computes automatically: F0 mean, minimum, maximum, and deviation standard (the latter as a measure of pitch variability). As a second Lindh 4 ………… … …………

Story …

R 2.9 72.7 11.2 0.05 4.64 4.24 1.64 7.81 1.87 0.87 1.43 1.58 3.59 3.71 5.35 0.53 11.36 15.91 23.25

Mächler, Mächler, Bolker, &

N 1.3 5.4 0.05 2.76 4.79 4.03 1.54 7.64 1.75 0.95 1.61 4.37 3.51 0.58 10.95 82.87 16.12 11.43 20.78 Bates,

; R 0.17 1.68 0.47 1.33 1.32 5.55 5.32 4.34 subjects subjects factor) on speech was 189.2 32.75 27.92 72.87 59.89 10.27 23.89 66.03 10.35

106.48 389.98 - values values are obtained by likelihood -

Mean

package

N (N, R) as a fixed effect, and both 4.8 0.17 10.2 1.68 0.56 0.97 1.23 5.23 9.67 4.42

33.44 28.11 72.91 60.12 22.62 63.76 105.44 383.64 190.82 lme4

Condition

(dB) (s)

(dB) lmer

(dB)

(syll/dur) (tot)

min max mean var. (Hz)

(Hz) rate (Hz)

(Hz)

as as random intercepts. P

pauses duration pauses

. . Each model included one of the 19 dependent variables

words

corrections

- var. min max mean

F0 PVQ Intensity Intensity Intensity Intensity Filled Silent Disfluencies Speech F0 F0 F0 Story n. Repetitions Insertions Interruptions Self Variable Main descriptive statistics

2 Participant listed in table and had and The effect of gesture restriction (within tested by running a total of 19 Linear Mixed Effects Models (henceforth LMEMs, R function Walker, 2014)

Table

FDR 0.0447 0.0289 0.0053 0.0237 0.0500 0.0342 0.0026

and and did

(est.= In other

. p 0.678 0.413 0.056 0.278 0.988 0.461 0.053

3.645)

04 - 0.67 0.173 3.645 3.754 2e 1.1752 0.5426 Chisq

speech rate was found

2.270, 2.270, S.E.= 2.096, Hochberg procedure, -

, , Chisq=

t 0.735 0.415 0.817 1.914 1.083 0.015 1.943 -

(est.=

6.39 0.15 0.506 1.106 2.589 0.455 0.728

Higher

C.I. 52 disfluency rates

significantly significantly changed between the two

0.33 0.33 1.85 0.449 0.004 0.456 0.034 ------1.277, 1.277, S.E.= 0.667 Lower

. . The results are reported with the adjusted

3.323) number number of words 0.23 0.122 0.186 0.212 0.397 0.667 2.096 S. E.

(est.=

, , Chisq= 0.09 - 0.003 0.362 0.088 0.325 1.277 2.270 the results of the 19 LMEMs. No significant effect of the ; Also, none of the Condition). Condition). The tests were then corrected for multiple

Estimates the full model against the model without the fixed effect of

Results

shows

Results of the LMEMs per dependent variable 2.3 3

3 - Filled pauses Insertions Interruptions Self corrections duration (s) n. of words Repetitions Variable Story conditions. conditions. As for F0 and intensity, speech was not affected by gesture restriction. Table not change in terms of Chisq=1.1752) 0.08, S.E.= 0.044 Table ability to gesture on any of the dependent variables words, stories told when participants were free to gesture were not longer in terms of duration (s) False Discovery Rate (FDR) critical values. critical (FDR) Rate Discovery False interest (i.e., testing via False Discovery Rate (i.e., Benjamini & Benjamini Hochberg, 1995) ratio tests of ………… …

………… 0.0184 0.0368 0.0158 0.0211 0.0132 0.0316 0.0395 0.0079 0.0263 0.0421 0.0105 0.0474 …

0.15 0.151 0.476 0.272 0.068 0.459 0.499 0.058 0.364 0.615 0.061 0.915 Hochberg correction

0.55 2.059 0.508 2.069 1.208 3.323 0.457 3.594 0.826 0.253 3.499 0.011

1.9 - 0.74 1.435 1.827 0.907 0.502 1.874 0.106 0.711 1.439 1.098 0.674 ------., Confidence ., interval: Confidence Lower 2,5%; Higher

C.I 1.92 0.085 0.264 1.241 0.006 3.786 0.055 0.809 0.005 0.009 0.643 24.814 not not gesture compared to when they

length, length, fluency and prosodic features

Story, 10.

did

0.39 0.165 1.715 3.293 2.195 0.009 0.717 0.545 0.124 0.192 0.544 - 12.135 ------

1.4 0.16 0.004 0.102 0.346 0.099 0.044 9.402 0.852 0.764 0.365 0.627 that speech ch content. However, previous findings have not Participants, 20 |

False Discovery Rate adjusted alpha levels (Benjamini

the speakers

“N” “N” (baseline) and “E” were recoded by contrasts (i.e., 0 was in between each

how FDR: 0.19 0.23 0.08 Groups: 0.07 0.002 0.037 1.619 0.693 - - - 1.035 6.340 0.525 0.688 - - - - when 200;

Discussion and Conclusions and Discussion

. results s

(Hz)

(Hz) 2.4 (Hz) (Hz) the

(dB) N. of obs: ,

(dB)

gesture

affect fluency and spee always been consistent. Also, little attention has been paid to the potential Previous studies have shown that gesture restriction can in some cases Thus did not change did 97,5% (R package confint). for multiple testing); Levels N). to equal being of instead level, mean var. Intensity (dB) Note: (dB) Intensity max Intensity F0 var. PVQ min Intensity F0 min F0 max F0 mean Disfluencies (tot) Speech rate (syll/dur) Silent Pauses

d not ects of on speech

The present

speech rate, effects could could compensate

ect influence of the As for

. previous research showed

(Arantes, (Arantes, Eriksson, & Lima, ngth ngth of the stories told in our participants

54 representative of longer speech excerpts. that that the stories produced were too short to drawing upon other cognitive strategies and by or 12 seconds long

memory or lexicon,

, , we expected a lower paced and more disfluent speech act act with the listener. The study takes into account a rms of speech rate or disfluencies rate this seems unlikely. The average le

fluency

. nger segments, speech excerpts should be at least around 9 seconds long specifically challenge for the inability to gesture 2018) It may be that, in the kind of task used for this study, which di (in R condition). Studies on the stabilization of speech rate in Brazilian Portuguese suggest that to give a representative idea of the speech rate of lo (Arantes & Lima, 2017) allow speech rate measures to be However, experiment is between 22.62 seconds (in N condition) and 23.89 seconds general planning difficulties. However, our evidence that study gesture restriction does has any detrimental not provide fluency in te possible explanation could be As for produced in the restraining condition, since that the inability to gesture can lead to lexical access difficulties or more and and intensity. However, the restraining results the use of show gestures on speech. no These results are significant commented features. acoustic and length, discourse for fluency, separately eff which effort was made to let speakers be comfortable with the task and naturally inter comprehensive set of measures related to temporal narrative fluency measures, and features, also focuses on acoustic measures related to pitch study study was aimed to gain further insight into the dir inability to gesture on speech prosody. An experiment was set up to elicit spontaneous story telling narratives in an ecologically valid setting, in effects of restraining gestures on speech acoustics specifically. ………… … - …………

). … but not

speaking. . To clarify this issue, related content (that is, even though gestures are eech in relation to, e.g., - d in N vs R conditions. d in R N vs , , and this may impact how Rauscher et al. (1996) et al. Rauscher

, Kita, Kita, 2000)

. . This would possibly require more

, , a future investigation could be Moreover 55 , , we did not find any difference in story that that any other movements participants did analysis analysis of disfluent sp king time. On the other hand, if the gesture stream

bility to gesture, as found in found as gesture, bility to Hostetter Hostetter et al., 2007) to to shed more light on the question of how the inability to

rmation rmation into chunks ready to be expressed in speech we cannot exclude (Emmorey & Casey, 2001; Melinger & Levelt, 2004, speech speech length measures Information Information Packaging Hypothesis, Kita, 2000) the content differencesthe content between speech produce is inhibited, this may lead speakers, for example, to just exclude (spatial) information from some their speech ( future work can complement speech length measures with an analysis of visually confirmed in words and longer spea As for length or in the number of words used. On the one hand, when gestures are restrained, speakers may rely more on speech the might speech need modality, to and integrate information that cannot be expressed speakers express spatial relations and action spatial content expressions might be the by the ina fluency, most affected, in terms of difficulties due to the fact that gestures have a role in packaging spatio motor info ( gesture can affect speech implemented fluency with an spatial and motor content expressions or an analysis of speech chunking in general. In fact, the inability to gesture can cause speech planning be embodied/simulated by other body movements e.g., legs, head, lips some positive role in the play of process still this could and Furthermore, restrained, restrained, with their heads, legs and shoulders had some speaking. In other words, the imagery and actions to be described can still role in the process of capitalizing on analytical thinking. as

e.g., the However, (Hostetter (Hostetter to gesture

recently recently found

that can interfere ), ), in fact, report that

truction (Krahmer & Swerts, 2007; ). ). Our impression is that empirical evidence coming from

Rimé et al. (1984 56 Walkington Walkington et al. (2019) , , we explored whether restraining gesture Dobrogaev Dobrogaev (1929) . . Our results, at a first sight, appear to be in

) . However, 9 and and legs.

, et al., 201

(i.e., (i.e., speakers might interpret the ins ures are restrained and are not overtly produced, valid acoustic analysis

and and head while sitting on their hands and we cannot exclude

y y with prosodic modulation, or they can even turn on speakers’ Hoetjes et al. (2014) , Harrison, t t polar opposites: the two kinds of instructions (i.e., encouraging and Hoetjes et al. (2014) and ference to F0, intensity and spectral properties their role for speech production. that when gest this applies to our data too: unsurprisingly, speakers still shoulders moved their that these movements could somehow replace actual hand gestures in forearms, shoulders, head prohibiting hands movements can increase movements in other parts of the body, including the eyes, lips, fingers and legs (this is also observed by slower, etc.). As a last note, asking speakers to sit on their hands, as we did, does not necessarily restrain them from moving other parts of the body, these might make speakers try to speak preconceptions differently to adapt to these a request to enact more, or speak in a clearer way, louder, or perhaps restraining gestures) can still impose additional cognitive load & Alibali, 2019; Marstaller differentl & Burianová, 2013) preconceptions about how speech with or without gesture should sound; partial partial contradiction with encouraging and the restraining speakers’ gestures should not be abovementioned considered studies. exac investigations investigations on speech prosody and gesture production suggests encouraging that the use of gestures has an effect on speech acoustics, with re Pouw has has any effects on acoustic properties such as fundamental frequency and intensity. Our results do not provide evidence for it and are with consistent As far as the ………… … ………… of

…

. byproduct shed more light to

a mere

allow

speech production to to produce a gesture (i.e., a

could and and

intend

57 can can be argued that even when people do gesture

e a gesture or movement, this does not discourse discourse length, fluency or F0 and intensity

spontaneous speech - propose, it

spontaneous narrative speech, we believe that further - speech and gestures are so closely related that not even

on semi

s does not affect mechanisms underlying

investigation the into To conclude, while the present study restriction provides evidence that gesture variations in semi gesture) gesture) and that by physically restraining speakers from using their hands can inhibit the speech effects/role have on gestures production. that actually Hoetjes et al. (2014) not actually visibly produc necessarily mean that they did not motor command can be there even though it is not overtly produced as a no no detrimental effects of gesture inhibition on speech production, math reasoning. The authors propose that gestures are and reasoning processes and do not cause any facilitation on it. By contrast, as geometry conjectures justifications can still be successfully provided with

………… …

………… 3219

- … se of peech U S 62(9),3204

, Effects of Encouraging Encouraging of Effects ). 2019 JournalofSpeech, Language, and HearingResearch ffects on ffects E

59 This chapter is based on the followingpublication: the on based is chapter This Encouraging the the Encouraging

estures: estures: G the Use of Gestures on Speech,In on Gestures of Use the Cravotta, A., Busà, M. G., & Prieto, P. ( G.,P. M. Prieto, A., & Busà, Cravotta,

- Abstract onfederate hancing the Encouraging Encouraging -

and and mean intensity metrics

ettings. ettings. The present study aims at 60

The study shows that asking speakers to use their hands

Previous Previous studies have investigated the effects of the inability to 20 native Italian speakers took part in a narration task in which Instructing Instructing speakers to gesture led effectively to higher gesture

ts: rections, rections, repetitions, insertions, interruptions, speech rate), and production. also also impact on F0 and intensity metrics. By showing that en gesture stream could affect speech prosody, this study provides further evidence that gestures and prosody interact in the process of speech werehigher in N. in E than Conclusion: while describing a story can have an effect on narration length, and can rate rate and salience; Significant differences were found for length (e.g., the (a) narratives had more discourse words in E than in N); (b) acoustic measures: F0 maximum, maximum intensity, Encouraging Encouraging condition (E), i.e., the participants were instructed to gesture while the telling story. Resul Method: they had to describe the content of short comic strips to a listener c in one of the following two condition conditions: (N), (1) Non i.e., no instructions about gesturing were given; (2) words and discourse length in seconds), disfluencies (filled pauses, self cor propertiesprosodic intensity). and (measures of F0, attention, especially in naturalistic s investigating the effects of encouraging the production of hand gestures on the following speech correlates: speech discourse length (number of Purpose: produce hand gestures on speakers’ prosodic features of speech; however, the potential effects of encouraging speakers to gesture have received less ………… … - - ………… and and

… ) among

(Gesture (Gesture Meadow, Meadow,

- (Krauss (Krauss et al., s s (e.g., electromagnetic . . [For a general duction

hypothesis, Kita, Alibali, Meadow, Meadow, 2012; Goldin - - (Asymmetric (Asymmetric Redundancy et al., 2017) Gibert & Prieto, 2013; Loehr, ć - Also, Also, as shown by recordings of

; facilitate lexical access . . Scholars have proposed various movements movements (using 61 Hostetter Hostetter & Alibali, 2008, 2019 express the speaker's mental simulation

(Krivokapi ;

conceptualization -

for - been shown that gestures and prosodic units are

(Cook, (Cook, Yip, & Goldin ndaries) (Gesture Hufnagel Hufnagel & Ren, 2018) Gibert et al., 2017; Esteve - - Introduction

irected positive role of gestures during speech production. estures estures and their role in the process of speech production (see d - (Esteve

3.1 . . These effects are still under investigation from different ; ; provide additional spatial information ow, ow, Nusbaum, Kelly, & Wagner, 2001; Ping & Goldin prominence and at bou concurrent concurrent speech and articulometry for vocal tract movements and a motion capture system for body body movements), final lengthening at prosodic boundaries extends to body movements (i.e., manual gestures lengthen during speech prosodically prosodically prominent parts of speech, or pitch others, accent 2012; Shattuck Gesture Gesture production is also very interconnected with speech production at the prosodic level. It has tightly synchronized from a temporal point of view. For example, gestural strokes or prominent parts of gestures (or gesture ‘hits’) tend to align with Mead 2010) perspectives disciplines. and of motor actions and perceptual states during speech pro as simulated action framework, reduce cognitive load & Chu, 2017; Kita & Özyürek, 2003) 2000) Sketch Model, de Ruiter, 2017) theoretical theoretical models based on a set of experimental findings which predict the self Gestures have been found to: conceptualization contribute to utterance planning and In the last decades an increasing bulk of speech research g has focused on co Church, Alibali, & Kelly, 2017)

) ral ure) ; see Malta or

, , and that Amanda ) in a number of some evidence that

triggers a clear direct investigated investigated whether

acoustic realization of e acoustic and/or visual

the occurring occurring word, showing that - ). ).

(Marstaller (Marstaller & Burianová, 2015 62 suggested suggested that, in fact, gestures can (Bernardis & Gentilucci, 2006)

Amanda Amanda goes to Malta speech gesture production engage a neu - Krahmer & Swerts (2007) ]. ]. Finally both behavioral and neuroimaging studies The vowels of the two target words ( . 2014) , , or in neither of these words, and a visual beat (a manual ( . . They asked Dutch participants to utter a target sentence Malta had an acoustic effect on the cooccurring word in terms of Malta occur with. or -

, visual beats have a similar Bernardis emphasizing & function Gentilucci as (2006) pitch accents. indicates indicates that if a speaker produces a visual beat (either a manual beat gesture, a head nod, or an eyebrow movement) this effect on the acoustic realization of the co frequency frequency (F0), maximum values of higher formants (F1, intensity F2, F3) (energy). and The results showed gestures that the production of duration beat (longer durations) and spectral properties (lower F2, F3). This beat gesture, a head or nod, an eyebrow movement) becould produced on Amanda were then analyzed in terms of duration, maximum fundamental (“Amanda (“Amanda gaat naar Malta’’: different ways by varying the distribution of th cues for prominence. For example, a pitch accent could be produced on Amanda producing producing a visual beat (head nod, eyebrow movement or hand gest on a given target word led to changes in prominence Studies Studies have shown that the production of gestures can have an impact on the prosodic features of cooccurring speech. There is hand gestures can alter the prosodic and spectral properties of the speech they co both speech production and co network connected to Broca’s area 2008 for review a Volta, & Dalla Gentilucci Wagner Wagner et al. have suggested that spoken language and arm gestures are controlled by the same motor control system overview on the synchronization between gestures and speech, see also ………… …

)

9 ………… In … vowel arm beat, - . They claim they found that nd nd F2 increased,

state state voiced - arm beat, two - meaningful meaningful gestures). They - Pouw, Pouw, Harrison, & Dixon (201 biomechanical basis

affect the actions of the muscles produced produced with meaningful gestures unique unique communication system. occurring word), occurring the F0 and spectral e two modalities are coded as an -

onation onation in terms of periodicity: a 63 can word (LAO) in four gesture conditions - s condition) condition) and either standing or sitting. They ., ., when the hand reached its lowest point) in iological iological evidence from like movements with high physical impetus (i.e., wrist , , meaningful/related, non -

occurring speech can also have a - extension extension and acceleration for initiating flexion). Also, performing movements when standing, as compared to sitting, increased were observed about 50 ms before maximum and extension 50 (i.e ms after the moment correspondence of with the highest impetus (deceleration for stopping found found that beat movement excluded) affected ph downbeat to upward movement phase of the beat seemed to temporally align with a peak in amplitude envelope and a peak in F0. Such peaks study study they asked the participants to phonate a steady (e.g., ‘a:’) while either moving their arms (one wrist beat) or not (passive co that hand gesture movement involved in expiration, and this could directly affect prosodic metrics of speech (e.g., contrasts in F0, as well as changes in amplitude). In their signal. Experimental phys shows that the relation between gestures and the acoustic realization of while intensity and vowel duration did not increase). Interestingly, this effect was only present when the gestures were meaningful; random arm that movement did is, not trigger a comparable effects on the speech (absent, (absent, present found that when the words were co related (i.e., withsemantically the co properties of the vowels were enhanced (specifically, F0 a integrated integrated signal that is directed by a their study, participants were asked to produce 3 common words (CIAO, NO, and STOP) and 1 pseudo enhance voice spectrum because th . - - ing using using different though this has

rease in imagery – speech production.

Hoetjes et al., 2014; the idea that the gesture during (e.g., (e.g., Dobrogaev, 1929;

Pine, Bird, & Kirk, 2007)

Also, Also, speakers were found to . found found that when speakers were

from definitions or pictures (i.e.,

. . As for speech length, studies . This might support (Graham (Graham & Heywood, 1975) or at least in the number of words used for

) was less semantically rich in expressing spatio restrain speakers’ gestures enerate target words Horbury & Guttentag, 1998; - from gesturing the speech used to describe motor tasks (e.g., (Frick

Hostetter, Hostetter, Alibali, & Kita (2007) (Rimé et al., 1984) es or describing drawings or objects less less fluent when speakers are prevented from gesturing while stori retelling picture naming task) when free to gesture than gesturing when prevented from With regard to speech fluency, studies have shown that speech becomes expressing spatial content not been corroborated in more recent studies (e.g., Hostetter et al., 2007; Jenkins et al., 2018) be more likely to g motor motor tasks) reported a general increase in the number of words used in the speech produced by speakers who were prevented (Finlayson from et gestur al., 2003) motor motor events. Also, spontaneous conversations produced by speakers that were prevented from gesturing showed a general dec content tasks (e.g., cartoon retelling, description of drawings or description of fluency. prevented how to wrap a package speech has been to The effects of the inability to gesture on speech production mainly have been assessed in relation to speech content spatial (i.e., relations semantic expression, imagery richness, content), speech length, as well as speech synchrony itself emerges from biomechanical constraints and not mechanisms. from neural only One of the methods used to investigate the potential effects of gestures on due to anticipatory postural adjustments. In showed that other merely moving words, arms affects the the acoustics study of phonation particular moments in time at the degree of entrainment of movement and phonation (but only for F0), ………… …

, , ………… and and

… earing Hoetjes n and, more n and, cognitive speech speech length

and and filled pauses).

extent. As for

speaking. Nonetheless, the effects of

65 but but does not contain any quantitative speakers had to give on instructions how

’s idea had been previously proposed by ed to a lesser more monotonous (in terms of pitch range) 2014) ( , , and especially when describing spatial content . . However, not all studies have confirmed these f of the participants having to perform the task while , , an early study which is often reported in the literature that gestures have on

is also, to our knowledge, the only study that has explored is by encouraging speakers’ use of gestures. Though

Hoetjes et al., roles Hoetjes et al. (2014)

Previous studies on encouraging gestures Krauss Krauss et al., 2000; McClave, 1998; Rauscher, Krauss, & Chen, 1996; ner, Malisz, & Kopp, 2014) actively actively eliciting gestures on properties have been investigat speech production and its prosodic the two types of instructions can in principle evidence theon effects have speechgestures on that productio provide complementary generally, can both contribute to shedding light on functional the A complementary way of exploring production the role of gestures encouraging in and restraining gesture speech use are not exactly polar opposites Wag does it provide nor used. the specific on methodology details analysis, speaking. Dobrogaev (1929) (e.g., when speakers cannot gesture. The study found no evidence of the effects of restraining gestures on pitch range; the speech data was perceptually and also it showed that listeners tested were not able to tell, by h speakers’ voice only, whether someone was gesturing or not while et al. (2014) the effects of restraining gestures on speech acoustics by whether investigating speech becomes to tie a tie, with hal sitting on their hands (other factors such as mutual visibility and previous experience were also tested). The study did not find effects of the inability to gesture on fluency (in terms of speech rate Morsella Morsella & Krauss, 2004) (Rauscher et al., 1996) findings: in Finlayson, Forrest, Lickley, & Beck, 2003; Graham & Heywood, 1975;

- - - d test - Vilà telling - speech is, to our Coromina, Coromina,

- showed that

gesture phase and and g story - to

Llanes - . . The study showed Lucero, Zaharchuk, & .

fluency Ravizza Ravizza (2003) educational found found that participants teste ,

ures or (b) beat gestures; or (c) instruction instruction condition (the longest - showed that observing storytellers

Giménez & Prieto (2018) -

fluency 66 Vilà 2019) lexical lexical retrieval (

also shows that gestures can help boost speech

) showed that asking Catalan speakers to produce lly lly related to also also found that asking speakers to perform non

h articulation. only only study that has explored the effects of encouraging instruction instruction phase (i.e., they found a significant difference - re condition and in the no from applied research in tapping) tapping) facilitated speech production in terms of resolution of

the tongue states and - santo santo (2014) of - children’s narrative abilities in story retelling. Moreover, Prieto, & Rohrer (2018 phase, when no instructions about Giménez, gestures Igualada, & were Prieto given. Also, who use beat gestures while telling a story improves the performance of gesture gesture production on narrative ability and that training children by asking them to produce beat gestures (or non referential hand gestures that associate with speech) durin improved children’s narrative performance and fluency in the post rehabilitation settings fluency and speec knowledge, the iconic gestu wereRTs in condition). the gesture found iconic Evidence they (a) had to perform either iconic gest had no instructions about gesturing. Their gestures results facilitated showed word production, that since beat Reaction successfully recalled Times words were shorter (RTs) in the beat for gesture than in the Casa referential gestures (i.e., beats) had a positive effect on word production. In this study, subjects were asked to recall words from definitions while respect to abilities idea asking participants to produce rhythmic meaningless hand movements (e.g., tip in a narrative task talked slightly longer in the instructed than in the no in mean story duration for the narrative data in the two conditions). With Parrill, Cabot, Kent, Chen & Payneau (2016) ………… … ………… ted The

it has …

corrections, ettings. - speech fluency

. to to hand gesture restriction

speech speech rehabilitation pauses, pauses, speech rate), as well as

e their speech intelligibility by

67 & Garcia, 2005) & Garcia, in a training phase can benefit fluency test phase. In - is to provide evidence, based on naturalistic data, usly usly investigated in relation word intervals, speaking time, total sentence duration, - (Garcia (Garcia & Cobb, 2000; Garcia & Dagenais, 1998; Garcia, Summary & research question Summary fundamental fundamental frequency and intensity. Considering previous findings, we expect that encouraging the use of hand gestures benefits intensity. F0 and on impact an has well as as and and only marginally in relation to hand gesture encouragement. These are: speech discourse length, fluency (number of filled pauses, self repetitions, insertions, interruptions, silent of the potential effects of encouraging gesture use on a comprehensive set of speech cues related to prosody. The object of analysis is features that have been previo To sum up, both encouraging and restraining the use of gesture appear to be valid ways to explore the role of gesture on findings speech need further production investigation especially but in naturalistic s aim of the present study chunking) chunking) Hustad 2000; & Dagenais, Cannito, causing causing an improvement of some aspects of the terms sentences of uttered inter (in speaking rate and phrasing strategy by triggering a more natural speech in terms of fluency was not produced gesture specifically phase, but in in the a post encouraging been shown that instructing adults with acquired dysarthria to produce hand gestures while speaking can enhanc in a posttest reading task, when no instructions to gesture are provided. These studies together point towards a positive effect of encouraging the use of gestures on speech fluency, though the speech that was evalua beat gestures while reading in L2

. - 5 Non partial

subject subject design (with a - to control for potential Encouraging gesture Study 1.

from the same university. As

than

68 ) in order to control for the unavoidable former students , , in which no instructions regarding how to

narration narration task in which the participants had to group of participants

(N)

gesture gesture production, as it might be the case that Condition

different

gesture , , in which participants were encouraged to use gestures

and and reported in Appendix. Similarly to Study 1, to control for Method

(e.g., Briton & Hall, 1995; Hostetter & Hostetter Hopkins, 1995; 2002) & Briton Hall, (e.g., , related differences in F0 values; and (b) (E)

2.2 involved a involved Materials Participants 3.2 were all from the Veneto region (age M = 24.2; SD = 2.9).

in subject factor: Study 2

Section Section than males males than For the narration task, we used the same kind of stories described in were recruited in the study for two main reasons, namely (a) to control for gender gender differences in females are more expressive and produce more gestures when speaking Padua Padua and 3 of them were compensation for their participation they fulfillment of course credits were or a free breakfast. either Only female participants given Twenty female native speakers of Italian participated in the experiment They Seventeen of them were undergraduate students at the University of presence of individual differences in gestures’ use saliency. and frequency in terms of types, gesture gesture while narrating were condition provided; and while telling the story. The experiment has a within with The present study used a watch and describe a set of comic strips in two different conditions: encouraging

5 ………… …

- …………

come … “ ). ). The subject subject - and and the with the ) ), ), fter the two instructions 2.2 , a as within up up cannot exclude - ilarly E condition Section Section N condition Sim

see condition

hence, participants participants were encouraged no no instructions regarding how hence, listener.

Study Study 1 ( a

(i.e., (i.e.,

ondition. as

69 (i.e., (i.e., the

in the N condition, with possible effects on subject subject design (with - participating as

this is because we believed that telling participants to

ure while narrating were provided; Procedure to to a N condition after having encouraged them to gesture was not h to beh to with comparable the E c

encouraging gesture condition ” - tudy tudy 2 followed the same procedure In the E condition the participants were given the following (translated from Italian): “Tell each story and use hand gestures to help familiarization trials plus other two familiarization condition trials excludes before the argument each that the N condition was enoug not trained condition condition could be more familiar with the task, more comfortable with the setting/the listener than their productions. However, the presence of two initial general other hand, we are aware that this experimental set possible order effects due to the fact produced that after the the E condition N is condition. always For example, participants in the E factor). factor). The order of the two conditions was kept the same (N, E) for all participants: back natural and it would lead to carryover effects between E and N. On the Encouraging gesture condition to use gestures while telling experiment has a within the story; comic comic strips set (2 extra familiarization stories + 5 target trials) in the Non to gest second half (2 extra familiarization stories + 5 target trials) in the S same research assistant story familiarization phase, the participants had to tell the first half of the presentations presentations which were counterbalanced across conditions (see section). next potential item effects, the target comic strips were shown in two orders of ) - 10 ×

uced uced per by a single consisted consisted in the Gesture rate was

In order to avoid gestural movements - Ekman & Friesen, 1969 . Non caused caused an increase in the condition. condition. All instances of co effortful part of the gesture that 1992) (Wittenburg, (Wittenburg, Brugman, Russel,

the participants. By this we made sure 70 (the most . The annotation criteria

s stroke

(Kendon, (Kendon, 2004; McNeill, remind the participants about the task. with the software ELAN trials) trials) lasting a total of 81.2 minutes (39.1 minutes in the N Assessment of the effects of the encouraging gesture prompt adaptors adaptors e.g., scratching, touching one’s hair; - ondition ondition to story story relative to the (Gestures/words*100). number of spoken words in the narration were excluded. The speakers produced a total of 2396 gestures (out which, of 1015 in the N and 1381 in the E calculated condition). per every story told as the number of gestures prod usually usually constitutes its semantic unit, e.g., two hands shaping together a rounded table (self speech gestures were identified and manually annotator annotated Klassmann, & Sloetjes, 2006) marking of all gestural entire data set. This was done to ascertain that instruction the in gesture the elicitation E condition had actually speakers’ gestures with respect to the N The amount of gesturing that was present in across the the speakers’ two narrations conditions (N, E) was quantitatively assessed over the of a total of 200 short narratives were obtained (20 participants target in the 42.1 E condition). and condition the E condition by the other half of conditions. across were strip materials comic counterbalanced that The experiment lasted approximately 30 minutes. Audiovisual recordings c potential item effects half of the participants explained half of the comic strips in the N condition, while the same comic strips were explained in you do so”. The instructions were kept visible during the whole E ………… …

- …………

… hereas, w

based on

left frame); . Upper (light grey) area: area: grey) (light Upper . , ure space (the lighter 5

salient gestures. - salient gesture (right). Both Both gesture (right). salient - Figure (

in a more central, higher and classification classification was done For the present annotation, a 2 McNeill, 1992) of the gest

in a less visually prominent area

The salient

’s representation of the gesture space : 4 Figure

(Streeck, 1994) representation (based on representation (based McNeill (1992)

, , each stroke was further classified depending on where it ’s representation of the gesture space, which is divided into d in the gesture space. rred to a "tuna can" appearing in one of the stories told in study. the told stories of one the in appearing a to can" "tuna rred salient (dark grey) area: gestures; non Lower Examples of a salient gesture (left) and and non (left) gesture salient of a Examples

5 salience Gesture space

4 Figure gestures refe Figure grey area), the gesture was coded as when the gesture stroke was produced When the gesture stroke was produced visually prominent area sectors sectors delimited by concentric squares. sectors version of in illustrated as used, was gesture was performe McNeill (1992) In addition, to assess whether instructing speakers to gesture also changed

; - P

(est. (est.

LMEMs prompt to = 10.723, . The

Gesture rate (est .

E) ,

(e.g, (e.g, those gestures l l model against the . henceforth,

( Salient gesture rate (NS) Each Each model included 4

- . and and had condition (N, E)

4 salient odels - Table

M encouraging

; non N and Table

6 te (S) and Non and (S) te ffects Salient Salient gesture rate E

72 gestures gestures produced per story relative to

Figure Bates Bates et al., 2014) ixed

M Salient Salient (NS) gesture rate. - nonencouraging ( it was coded as

inear g the arms along the legs or on the armrests Salient Salient Gesture (S) rate was computed per every L package;

). lme4 in per condition

lmer , right frame 5 Gesture rate (G) and Salient gesture ra

6 he effects of encouraging gestures on gesture rate and gesture salience Figure Figure SE = 0.794, p < .001) were significantly higher in the E condition than values values are obtained by likelihood ratio tests of the ful model without the fixed effect of interest (i.e., condition). = 4.134, SE = 0.708, p < .001) and was was tested by running 3 R function one of the 3 dependent variables listed in as a fixed effect and both story and participant as random intercepts. T the number of spoken words in the narrative (Salient gesture/words*100). The same was done for Non workedgesture shown in well, as performed while keepin ( story told as the number of salient (the lower darker sector), ………… … ………… on

…

6.589, 6.589,

- s p <.001 <.001 <.001 N. of tudy : : (1) Story (1) speech

Chu & Kita,

: . Each model model Each . in (est. (est. = coustic

Chisq 31.217 125.57 80.71 kers to use more a

(N, E)

10.13 t 5.838 13.51 - the following measures

Bolker, & Walker, 2014) luency and and luency , , leading spea and and disfluency measures; (3)

f 5.311 Higher 5.526 12.283 - C.I Salient Salient gesture rate Similarly

performed exactly as ).

values obtained by likelihood ratio tests of the full - 73 Non gesture space. gesture 7.868 2.2

Lower 2.742 9.162 - Bates, Mächler, was as as found in previous studies, s

Meadow, Meadow, 2010) - Section Section S. E. 0.708 0.794 0.65

see

data data analys Salient gesture rate (per 100 words).

effects for models the effects of condition on Gesture rate, Salient - - each story told in the two conditions 6.589 the

Estimates 4.134 10.723 - in ndition, ndition, and conversely

Linear mixed Statistical analysis Data analysis: Transcriptions,

seline) and “E” were recoded by contrasts. P recoded seline) were “E” and 4 Models: R function lmer in lme4 package (

Salient - 200; Groups: participants, 20 | Story, 10. C.I.: Lower 2,5%; Higher 97,5% (R package confint). Levels

The data analysis focused on a total of 19 variables of interest duration (in seconds); (2) Number of words per story; (3) Repetition rate; discourse discourse length measures; (2) fluency measures. acoustic In this study, Restraining gestures ( were taken 2011; 2011; Cook, Yip, & Goldin higher a involve also that gestures To sum up, the quantitative analysis of the gestures performed speakers by suggests the that encouraging speakers to use telling the their stories worked well hands ( while included Condition (N, E) as a fixed effect and both Story and Participant as random intercepts. intercepts. random as Participant and Story both and effect fixed a as E) (N, Condition included obs: (ba “N” model against the model without the fixed effect of interest (i.e., Condition). gesture rate Non gesture rate Note: Variable Gesture rate Salient SE = 0.65, p < .001) was higherwas p< .001) SE = 0.65, in the N condition. Table gesture rate and Non in the N co ent ent (PVQ);

4.69 3.17 2.18 0.75 1.31 1.58 3.89 4.27 5.97 0.62 11.08 68.86 17.57 10.36 0.05 2.46 10.25 24.74 SD

N 4.79 3.51 1.85 1.16 1.45 1.49 4,12 4.69 6.23 0.53 9.92 66.36 18,61 9.87 0.05 2.57 8.17 21.35 correction correction rate; (7) Filled -

74.61 61.36 1.95 0.41 1.02 1.45 6.25 7.76 11.08 4.21 105.25 414.43 191.94 35.03 0.18 27.84 25.23 67.19

Mean

74 N rate; rate; (6) Self

73.36 60.51 1.47 0.54 0.86 1.2 6.7 7.61 10.78 4.24 105.44 391.86 191.49 33.88 0.18 27.82 23.44 62.1

/dur)

Table

corrections - shown in F0 mean F0 (Hz)mean F0 var. (Hz) PVQ (dB) min Intensity (dB) max Intensity (dB) mean Intensity Filled pauses Silent pauses Disfluencies (tot) Speech rate (syll F0 min (Hz) F0 (Hz)max Variable Story duration (s) N. of words Repetitions Insertions Interruptions Self

Main descriptive statistics

(16) Minimum intensity; (17) Maximum intensity; (18) Mean intensity; (19) Intensity standard deviation. The main variable are descriptive statistics per Table pauses pauses rate; (8) Silent Pauses rate; (9) Total Disfluencies rate (including 3, 4, 5, 6, 7); (10) Speech rate; (11) Minimum F0; (12) Maximum F0; (13) Mean F0; (14) F0 standard deviation; (15) Pitch Variation Quoti (4) Insertion rate; (5) Interruption ………… …

- ………… lme4

.E .E =

… in that was

values are - lmer

(N, E) as a fixed 1.31

Number of Words . The results are

). ). The tests were then Maximum Maximum F0 1.39 (R (R function Condition dness dness that can be heard in subjects subjects factor) on speech -

had Condition

(est. (est. = 1.254, S.E = 0.297, p 10.86

LMEMs LMEMs as as random intercepts. P and

5 Each Each model included one of the 19 75 . f condition for 10.84 (est. (est. = 0.846, S.E = 0.15 p <.003, FDR),

Table Participant

Benjamini & Hochberg, 1995)

and and Maximum Maximum Intensity

listed in Mean Intensity Mean Story the results of the 19 LMEMs. There was an effect of

Bates Bates et al., 2014)

in any of the disfluency measures, nor there was any effect on Results . . Finally, as for the acoustic measures related to F0 and

Intensity var. (dB) var. Intensity

re both higher in E than in N. While the results on the Maximum shows

6 3.3 Condition 1 dB corresponds to the smallest change in lou which we F0 are quite impactful (a mean difference of 22 Hz), a more subtle effect was found for intensity. However, we should note that a change of about intensity, there was an effect o higher in E than in N (est. = 22.578, S.E=8.302, p <.008, FDR); and an effect was found for <.005, FDR) and 1.945, 1.945, p <.011, FDR). As for the measures of fluency, there was no effect of Speech Rate Table condition on speech discourse length, specifically on that was higher in the E than in the N condition (est. = 5.09, S reported with the adjusted False Discovery Rate (FDR) critical values. critical (FDR) Rate reported with Discovery False the adjusted obtained by likelihood ratio test of the without full the model fixed effect against of interest the (i.e., model corrected for multiple testing via False Discovery Rate (i.e., Benjamini Hochberg procedure, package; see dependent variables effect and both The effect of gesture encouragement (within was tested by running a total of 19

0.045 0.005 * 0.024 0.026 0.042 0.039 0.037 0.050 0.008 * 0.034 0.018 0.021 FDR 0.013 0.011 * 0.016 0.029 0.032

0.799 0.001 0.219 0.279 0.73 0.649 0.538 0.878 0.007 0.489 0.109 0.15 p 0.017 0.009 0.04 0.318 0.377

0.065 17.043 1.509 1.17 0.119 0.208 0.38 0.024 7.272 0.478 2.574 2.08 Chisq 5.73 6.753 4.237 0.996 0.779

1.081 0.615 0.153 0.997 0.254 4.221 1.228 - 0.345 0.455 - - 2.719 0.69 1.606 1.443 t 2.407 2.617 2.066 - 0.881

iable 0.173 1.837 0.63 0.368 0.971 1.586 0.065 2.259 38.904 1.761 2.554 0.013 Higher 3.256 8.913 0.931 0.128 0.518

0.393 0.197 0.146 1.267 0.682 0.99 0.123 2.641 0.846 0.257 0.002 0.133 - - - 6.251 - - - - 0.67 - - - - - C.I Lower 0.329 1.268 0.023

0.715 0.004 0.078 0.297 0.655 0.048 1.247 8.302 0.663 0.133 0.182 0.197 0.416 0.42 S. E. 0.745 1.945 0.231

0.029 0.191 0.132 0.45 1.148 0.005 0.0198 1.254 0.298 - - 22.578 0.458 - 0.16 0.242 - 0.145 Estimates 1.792 5.090 0.477

(Hz)

Results of the LMEMs per dependent var

corrections

- mean

Intensity max max Intensity (dB) PVQ min Intensity (dB) F0 max (Hz) F0 F0 var.(Hz) Speech rate (syll/dur) F0 min (Hz) Silent pauses Disfluencies (tot) Interruptions Self Filled pauses N. of words Repetitions Insertions Story duration (s) (Ladefoged, 2003) (Ladefoged, Table Variable a sound booth (e.g. a change of 5 dB corresponds to doubling the loudness ………… … …………

…

0.003 * 0.047 –

effects of Hochberg – 0.001 0.806

prosodic prosodic 29.459 0.06

comprehensive set of

Pitch Variation Quotient

variables variables that showed a 4 5.652 0.245 PVQ PVQ =

explore the 1.14 0.215

condition. 77

0.167 False discovery rate adjusted alpha levels (Benjamini 0.552 -

participants, 20 | story, 10. Confidence interval (CI): lower, across

fundamental frequency. frequency. fundamental

FDR: = number of syllables produced in each audio file divided by the file's

F0 = 0.15 0.097 ; Groups: 200; syll/dur Condition

jittered boxplots refer to the

They show the full range of variation (form minimum to maximum), 4

0.846 0.024 the Discussion and Conclusion and Discussion Box plots representing the variables that showed a significant effect of

7 7 3.4 N. of observations:

Figure

Condition (N, E). the likely range of variation (the interquartile range), and the typical value (the median) median) (the typical value the and range), interquartile (the variation of likely range the narrative narrative speech. The study took into account a The aim of the present study was encouraging the to use of hand gestures on naturally elicited spontaneous In effectsignificant of 2.5%; higher, 97.5% (R package confint). Hochberg correction for multiple testing); An asterisk denotes significance after Benjamini correction; Levels(baseline) “N” by contrasts recoded and “E” were (i.e., each level,0 was in between N). to equal being of instead duration (De Jong & 2009) Wempe, Intensity var. var. Intensity (dB) Note: mean Intensity (dB)

. some

higher Parrill et to tell each made to let

. Let us now

, , related to : : (1) production of speech

Also, a more thorough

more words ? have a facilitating role in

issues: Does encouraging the louder

, , little attention has been paid esult is coherent with positive effects that encouraging

gesture gesture salience

Lucero Lucero et al., 2014; Ravizza, 2003)

78 and and ; ; and (3) peech chunking , in some cases,

prosodic prosodic features of speech ly ly valid setting, and effort was and and s -

gesture gesture rate as well as with the in terms of number of words used; (2) acoustic

, especially in naturalistic settings. in naturalistic especially , exical exical retrieval via definition, objects, drawings or motor lating lating to discourse length and fluency, and acoustic measures ’s findings, prosodic prosodic planning ic features , , when instructed to gesture, speakers used affect use use of gestures enhance the semantic density and richness of speech? Does it al. al. (2016) gestures have on lexical access (e.g., We believe that a more thorough content analysis of the narratives could contribute to shedding light on the following story. story. The increase in the number of words in significant, the is E rather modest condition, (approximately albeit 5 words) and it should interpreted with care. In spite of this, this r be fundamental frequency maxima each of separately. these on comment results First effectively increasing modest changes in longer discourse The present experiment was set up to elicit narratives spontaneous in an story ecological telling speakers be comfortable with the task and naturally listener. interact The with results showed the that encouraging speakers to gesture, while the process of speech production. However to the effects that encouraging gestures can have on both prosod fluency and and and tasks (e.g., l task descriptions, story retellings, vocalizations in the lab). These studies have suggested that gestures can relating to pitch and intensity. intensity. pitch and to relating Previous studies on the role of gestures in speech fluency and have used prosody a variety of methods and heterogeneous experimental settings measures re ………… …

- ………… of in on

the the

Kita Vilà been

(

2007; 2007; … to .

er, gesture 4

- gestures working terms speakers speakers evidence found

never in on speech gestures

encouraging phase for

The g their

Howev

aximum were

, more

had Chapter .

on that

detrimental in

2012

fluency, encouraging training be 2018)

conditions

fluency al., a

encouragin

demand is a highly perceivable not producing production: al., that

et expected two

(e.g., (e.g., Finlayson et al., 2003;

discussed speech

et the

speech the may

on Cook about which

effect speech on

possible

features in the speakers’ narratives initially

gesture

)

further

an 6 see is

79 to gesture be effect it with increased

We of

between

Coromina

- to (but

thinking gestures

will

Table case, have y intensity rate.

Llanes

speakers our ; evidence

and and

beneficial (Hostetter (Hostetter & Alibali, 2019; Kita et al., 2017;

interfered aspect difference fluency

might

attention In

a . . This would be needed to reconcile the overall

speech

no 2018 find Hz (see

This consciousl encouraging this

have

and have not of

was

and favoring drawing

encouraged

been

Prieto, did

might rates specifically.

F0 maximum not &

There would that memory).

that .

have

influence talking,

gesture

approximately approximately 22 difference. In addition, the F0 maximum variable is directly related to (that (that is, F0 maximum, Intensity mean significantly higher in the E than in the N condition). As for F0 maximum, and Intensity m the results showed a difference between the two conditions of working Third, the results showed increased that encouraging participants to gesture while memory, showing direct addressed to might gestures studies Giménez Second, fluency disfluency Jenkins et al., 2018) findings with the predictions production frameworks made by the & 2003) Özyürek, major speech studies studies investigating the effects production, which have yielded mixed results of restraining gestures Graham & Heywood, 1975; Hoetjes et al., 2014; Hostetter et al., investigation would allow a comparison of our results with those from . - - (see non / the two like - the results

modulation modulation

and and future

beat is of since ech. Indeed, the ,

. one should not expect

, a future analysis of our

speakers’ speaking styles

2007) e et al., 2019) (

mentioned

speakers’ to gesture could switch

influenc how speech produced with bodily

) showed that the inability to gesture 80 posals posals that defend a biomechanical , Harrison, previously . . In this direction, Also, Also, future studies might explore whether interact interact with speech prosodic features

and and thus encouraging as

Krahmer &Swerts Pouw ,

lso, gestures gestures of different strengths affect speech to A ning ning gesture experimental setting by Hoetjes at al.

stures e.g., ( ge

Hoetjes et al. (2014 . However, to to this, our results might seem in disagreement with those prosodic prosodic changes evident in the current study may not be - on speakers’ preconceptions about expressions should be like and and prosody experimental settings are not exact polar opposites polar opposite results. reference obtained in the restrai (2014). Indeed while speaking does not have any detrimental effect on pitch encouraging encouraging gestures on speech is needed: a joint analys from the two paradigms (i.e., restraining and encouraging gestures) can help to clarify the interactions between gesture and speech prosody. With representational in differently, discussed as We believe that further work on the effects of both restraining and Bernardis & Gentilucci, 2006) data could explore if different degrees over time. different types of gestures e.g., representational vs investigations investigations are needed to further explore the existence motor control system of that controls the production of both modalities, a i.e., shared gestures and speech, which can mutually enhance/reduce each other research is needed before any firm conclusion can be drawn. Nonetheless, our results are in line interdependence with between gestures and the acoustic pro realization of the co occurring speech relevant relevant in the expression and detection of emphatic spe acoustic apparent to an everyday listener in a naturalistic setting and more pitch range or F0 variation and amplitude measures, which are very ………… …

, ………… …

Although

. These need

.. (Briton (Briton & Hall, 2015) More generally, gesture gesture while doing a provides evidence that a

has been shown that high , it

This line of research, building on ample

ion and speech prosodic modulation

81 and and low extraverted individuals may rely on

(e.g., (e.g., in the case of bilinguals, Dewaele & t t other individual differences, e.g., personality traits and . Also, high speakers speakers might interpret the instructions as a request to be considered as potential side effects playing a role in our study. side role inbe a our effects potential considered playing as is in place, in a naturalistic setting. previous successful clinical attempts that used gesture encouragement as a To conclude, despite some limitations, our study relationship between gesture product which the results can be generalizable (also to different cultures). Also, it could be argued that encouraging speakers to different task (e.g., explaining a motor task) might yield different results. beThis may worth further too investigation. the experiment) we think that, in a future study, a data collection takes that into accoun cognitive and linguistic abilities, might help to evaluate the extent to Göksun Göksun et al., 2013; Hostetter & Hopkins, 2002; Hostetter 2012; Kita, 2009; & Nicoladis et al., 2018; Potthoff, O’Carroll et al., we controlled as much as possible for linguistic and gender differences (only female speakers from the same regional area in Italy participated in physical physical properties largely depend on individuals’ personality linguistic traits, cultural, and gender differences cognitive abilities 1995; Chu et al., 2014; Gillespie, James, Federmeier, & Watson, 2014; extraverted extraverted individuals Furnham, 2000) gestures for linguistic fluency individual to differences in different gesture production extents. in terms of rate, type and does not take into account individual differences that might play a role in speech and gesture production. For ex extraverted individuals have naturally more fluent speech than low enact more, or speak in a clearer way, or perhaps slower, etc to A possible limitation of our study that should be acknowledged is that it For example, (Garcia (Garcia & Cobb,

82 direct influence of gestures on speech e ground for therapies in communication

and and from a more theoretical perspective, this

provide a fertil More broadly, , can

study study opens new questions on the deserve further that investigation. production 2000; 2000; Garcia & Dagenais, 1998; Garcia et al., 2000; 2005) Hustad & Garcia, disorders. way to improve the speech of individuals with dysarthria ………… …

- …………

, the … might baseline focus on

less clear no gesture

nstructions , (Hostetter (Hostetter & to

heterogeneous

, their Remarks More generally this seems to be the

. to explore the role of rich and

the two paradigms

a respectively,

, In this chapter, we

. Methodological and Encouraging and Encouraging : personality traits

, in general,

a a valid way first glance, 83

, for instance, on how the i a

Though moving to some extent. Likewise, when At individual individual cognitive resources expected to lead

besides hand gestures

depend

affect individual speakers’ behavior in slightly their

Gestures

gesture. ould Restraning and ure speakers might exhibit

can can be considered since they are

. This could s increased e For example, it could be the case that even when sitting on

opposit

m gesture gesture production Alibali, 2019; Marstaller & Burianová, 2013) pattern pattern of multimodal cues same instructions c different ways are interpreted by each speaker, their cut picture. their hands, people are still encouraged to gest see produced vs case, a closer inspection of the data can give a more detailed Asking Asking speakers to sit on their hands or encouraging while them speaking to gesture gestures in speech production

2 in to

d that n some I n, n, 1999;

inhibiting .

exhaustive necessarily

(Morsella (Morsella & carried out (Jenkins et al., not

ed in Chapters still still moved their explore them not to move their z Dobrogaev, Dobrogaev, 1929; As a consequence,

. sit on purposefully oven oven mitts or Velcro

to analy

, we ) 6 wear ( expected effect of data (Beattie & Coughla

; rather to

the ) the ;

2 d arms (as (as reported in

; ( hold a rod or wear an apron with ha

84 inspection inspection of our data suggeste to

their (Hostetter (Hostetter et al., 2007; Pine et al., 2007; asking asking participants (1) )

grip the bottom of the seat inform future work and methodological 5 (

to ; and and legs

fold ed, partial immobilization of the body was ) (Emmorey & Casey, 2001; Hoetjes et al., 2014;

board , and 3 while sitting on their hands / (

to . ; ), ), a visual ) from moving other parts of the body, e.g., the 2 . . Inde 4

in this chapter we do not aim to bring (Finlayson (Finlayson et al., 2003; Rimé et al., 1984) ; ( by providing a brief overview of some qualitative and and and legs it was reported that these methods do

, onetheless, onetheless, we noticed that participants Horbury & Guttentag, 1998) - eir potential Restraining gestures Restraining

. N . head th

1 (Chapter , sit sit on their hands participants

Therefore, (Frick

however, 4.1 or . (Rimé et al., 1984) (Rimé et al.,

3 tudy tudy S gesture use shoulders move In asking speakers to sit on their hands forearms, shoulders, head Hoetjes et al., 2014) shown to lead to an increased motor activity in body parts left free to devised armchairs cases, restrain 2017) Özer et al., 2017) Graham & Heywood, 1975) Velcro Krauss, Krauss, 2004; Rauscher et al., 1996) gloves attached to a table Walkington et al., 2019) In previous studies, restraining speakers’ gesture has been different ways. For example, by hands while wearing dummy electrodes on hands/arms highlight approaches quantitative quantitative aspects that we observed in and experimental evidence on these issues these phenomena ………… …

: ………… in IT: ( by,

, p.

fairly … actual imposed proposed

the s instructing

reduced d For example . In one case, a

ce a gesture even of increasing the On On the other hand, ). can be (McNeill, 2017 This is in line with hard .

. It’s It’s clear that I’ve just speech production be

that they were In addition, a

to to produ on a a motor command can be

commented spontaneously the risk

, It’s It’s going to be hard for me

intend ds ds rather than simply impact of them

additional cognitive load

However cannot cannot / 85 . again some

, (Hostetter (Hostetter & Alibali, 2019; Marstaller & “even if, for some reason, the hands are not not externalized, the imagery it embodies can

would would My My poor language skills… people may still

, ) EN an an actual movement (i.e.,

( . right after having told the story in the restraining

. This sitting sitting on their hands while speaking

ith setting setting is cognitively demanding and interacts with individual dition: dition:

observation that realizing ory skills not to gesture to not So So I can’t move them? No, I just asking speakers to sit on their han

kind of cannot cannot exclude that these movements somehow replace )

. Hoetjes et al. (2014)

)

EN explanat gesture con ( Quindi non posso muoverle? No per me participant è un explicitly dramma complained about her poor vocabulary and comfortable w after receiving the instructions that speaking without moving their hands would In our study, we aimed to avoid as much as possible unnatural postures and annoying restrictions of any participants type. commented, When debriefed on offline, the our one hand, different ways across participants load resulting from having to remember not to gesture e.g., them this working memory capacity Burianová, 2013) 78 A related issue concerns the possible to speakers by the inability to gesture. It cannot be excluded, in fact, that there even though not overtly produced as a gesture) McNeill’s restrained and a gesture is still be present, hidden but integrated with speech” hand hand gestures in their role for speech production. by when not we

- to In be

al., erly the ). ). - These which of

- , . , speech prop could could alternate the right herself a…. a…. come

i vestiti to

for this Hostetter et , , second language Meadow, Meadow, 2010; Chu see - the speaker ; it’s it’s so hard without hands!... hile searching for involve combined

It It jumps in the suitcase and

it was w

)

… Coromina Coromina et al., 2018) - EN ( , possibly informing the choice of Giménez & Prieto, 2018) would -

st st provide some insight on how e.g., e.g., kneading 86 (Beilock & Goldin , A possible explanation ; Llanes (Vilà , can inspire can questions. future , 9 (Broaders et al., 2007) gestures Entra Entra nella valigia e comincia a…. not semantically rich enough Though no conclusions can be drawn from aws). aws).

… è difficile senza mani!... toccare solving - TOT state. Furthermore, the speaker eventually (IT: m

. seconds seconds of speech)

. . ah . interesting to note that to the another another case, a speaker experienced a long tip that was

d how how do you say it? Starts to learning math le

yet

: : la mia proprietà di linguaggio… si vede che mi sono svegliata (Baills (Baills et al., 201 the clothes , about about 15

( In Encouraging Encouraging (IT

) which starts… Finally, Finally, it is . . comincia aa.

her inability to gesture for this problematic lexical access few examples, they at lea . 4.2

. touching … the motor simulation of the action was challenged by the inability to fluency fluency and narrative abilities studies have shown that gestures have a beneficial role in thinking, gesture gesture in order to explore the effects of encouraging the use of gesture on activities such as problem & Kita, 2011) pronunciation Researchers in previous studies have explicitly instructed participants to participants cope with the instructions experimental settings. Moreover, these are interesting cases that, framed in accounts the theoretical major blame she overtly commented on. these gesture, selected a word describe the action (touching vs. 2007) this case, the action to be described movements of both hands (p that walking while kneading on clothes): starts starts si dice. da da poco… tongue state word to describe a cat in the act of pawing inside the suitcase (i.e., woken up… ………… … ………… … gesture

speakers ; ; (2) the ing speakers to ; ; (3) the addressee’s Parrill, Parrill, Cabot, Kent, Chen, & own that gesture rate, together

87 and and physical properties largely depend (Bavelas, (Bavelas, Gerwing, Sutton, & Prevost,

, t t the same time, the instruction does not rforming three different discourse tasks (i.e., a

). ). Moreover, individual differences in gesture ; to gesture generally causes an increase in the

rate, type

(Briton (Briton & Hall, 1995; Chu et al., 2014; Göksun et al., contrast contrast with previous findings, mentioned above, that . . The study compared the differences in gesture rate and (Gerwing & Bavelas, 2004; Holler & Wilkin, 2009) esture production may be influenced by a combination of ck (e.g., gesture rate lowers when addressees are less attentive conversation, conversation, spatial problem solving, and narration). In the study, ver, g - e gender differences 2013; Hostetter & Hopkins, 2002; Hostetter & Potthoff, 2012; Kita, 2009; (Jacobs (Jacobs & Garnham, 2007) production in terms of on the individuals’ cognitive abilities, personality traits, cultural and interlocutors interlocutors’ (mutual) visibility 2008; Bavelas, Kenwood, Johnson, & Phillips, 2002) feedba other factors. For instance, it has been sh with gesture type and gesture physical forms (size, salience), can change and be adapted depending on (1) the shared knowledge between show that the instruction to rate. gesture increasing gesture should at least contribute to How gesture gesture will not always work (in the sense that it might not lead to produce more gestures) seem to impact on the type of appears gestures to be produced. in In sum, the study quasi the instruction to gesture did not change gesture rate across the different discourse tasks, suggesting that instruct or gesture type on the way they gesture across genres is Payneau, (2016) gesture type of participants that had been and had not instructed to gesture while pe been explicitly instructing instructing participants participants’ gesture rate. Nonetheless, to our knowledge, the only study that has directly focused on the impact of encouraging speakers to learning, remembering, and speaking. As well, they have shown that

. . , e

iv are

rate, , for a ( use of speakers’ that g role shifts gesture can time enactment gesture - intentionally ns

representational ing - it may well be that

, enactments allow the

, increas should should be be worth assessing and forms of

more general Also

of instructing speakers to . . These studies suggest that and and can be found in spoken effects of explicitly asking can

This may result in the which

, experience of the original

gesture gesture rate, type and salience salience

impact ., ., 2015) including of the instruction to gesture on factors 88

interpret the instruction as a request to face interactions

and and -

of gesture produced and on more general

witnessing the event in real to that instructing participants to -

the potential ng gestures on gesture production visual enhanced bodily expressions should be like impact switches on some

types multimodal multimodal cues

clearer gestures (i.e., (i.e., representational vs. non

. , the potential general type as they are gesture gesture types may well be Ferrara & Johnston, 2014)

we explore

, it

in experimental settings. ure the participants may

are well documented in sign languages (e.g., or produce ,

Moreover Stec, 2012) Stec, to feel s

affecting . e

erefore, see Effects of encouragi

more as gest Th /action/character. In face

also

aspects aspects of how gestures are produced, intended and interpreted in the

hen setting up an experiment or data collection that requires explicitly nactment or example In this section speakers to gesture on the controlling for controlling assess the possible such gestures) gesture on more general review W asking participants to gesture while speaking, it might be important to interlocutors interlocutors a event direc addresse different multimodal strategies, E constructed action languages as well. They are bodily mimetic demonstratio F enact designed for the benefit of the addressee. impact all of these aspects in combination, e.g., by and the prompt to gesture preconceptions about how all of these features together, namely, key wild. Nicoladis et al., 2018; O’Carroll et al ………… …

………… In all the

). ). ually

3 … and

man their further gesture gesture rate respect to the so by carrying -

do and and

The analysis showed s We Study Study 2 (Chapter ). ). . The annotation criteria 5 . hat represent semantic the data was stroke

al we have further annotated

Figure

. increased speakers’ enactment rformed in both conditions (N, E) ) (Wittenburg et al., 2006)

and and 6 Representational and Non gestures t

4 89 . . To do so, were identified salient salient depending on where they were -

i.e., ( Figure (N) that that the gesture elicitation instruction had , reporting the data collected for

Figure

non

f ,

3.2 ed we present here,

vs 3.2 with a focus on Box plots encouraging and Encouraging gestures, N, E).

- 8 condition ascertain ection salient speech gestures

S (Non - (see Section Figure we

, (see Representational

representational gesture rate (per 100 words) in the two conditions conditions in the two representational gesture rate (per 100 words) 3.2 additional additional analysis o additional additional analysis encouraging - an the on between and salience In data by distinguishing the gestures pe distinction distinction into performed that encouraging gestures effectively N annotated with the software ELAN instances of co consisted in the marking of all gestur out Section actually caused an increase in the speakers’ gestures with multimodal strategies,

- - )

For The ) by . Linear

enact Non

6 . 3 ) and and had

values are iconics

without to two -

Each model . and the rate of make use of gesture (e.g., (e.g.,

made a bodily of the story by , , p <.001) gesture rate

s was tested via

w that expressed in the co 0.586 a narrative task itself palm up sho Study Study 2 (Chapter gesture Bates et al., 2014) participants ,

8 higher in the E condition the speaker character

he model without the fixed effect of

, S.E = is from or

suggests suggests that drawing speakers’ . . We found a significantly higher Representational (NR) Figure - 9 4.776 the referent

package; see there was no significant difference in which

This Non

, 90

For example, a also on the types of gestures speakers . n n to gesture also led speakers the data representational gestures

subjects subjects factor) on

than than they would naturally do - or .

lme4

Figure rate , in in but (est. = 1.2

) (i.e., (i.e., gestures whose handshape, trajectory y recall (N 3

It is worth noting that lmer s

annotated annotated any moment

hen encouraged to gesture, see Section

stration of some actions , to gesture. n do do not visuall whether the instructio enactment have further speech found found in Chapter That is, w

as . of the analysis, as shown in than in the N condition e we

(as (as s

location Representational Representational gestures instruction -

on The effect of gesture encouragement (within

ncluded: Representational (R) gesture rate obtained by likelihood ratio tests of the full model against t Condition). (i.e., interest Mixed Effects Models (R function i Condition (N, E) as a fixed effect and both Story and Participant as random intercepts. P and and shoulders, as illustrated in marking depiction/demo means of facial expressions, and combined movements of the hands, head compared with this is other worth speech and furthertasks investigation. To evaluate more, produc more representational gesture any may be more likely to elicit more between the two conditions attention on gestures impacts not only on gesture rate and their use space of result representational gestures (per 100 words) stories N Representational Representational gesture and occurring information via handshape, trajectory, or location) 6 ………… …

………… or Si Si

and and its its This . F

, … (E) . zation

| activate We believe

the cat with its eyes wide eyes its with cat the

Arriva il gatto all’improvviso all’improvviso gatto Arriva il E lei (la gattina) si volta così: sicosì: gattina) volta (la E lei (EN)

For example, participants more general involvement

(B) (B)

| (A)

(N=7; E=44; t= 3.944, p<

And the cat suddenly arrives with with arrives the suddenly cat And

ay ay be worth consideration benefit benefit of the addressee m lead lead to (EN) and the bigger cat is… is disgusted is disgusted bigger cat is… and the haughty ;

) it walks away and glares at him. at glares and away walks it speculated speculated that gesture of different

91 e production can give a clearer idea of EN eft to right: right: to eft l earlier stage of the conceptuali (EN) gesture E condition . like this:

an From From the (agli uccelli) (agli la gatta sgrana gli occhi; gli occhi; sgrana gatta la

away si schifa; ( schifa; si

at in effects on the speech produced, due to the fact

… (C) of more representational gestures or instances of | effects on gestur

the cat turns eing aware of the impact of explicitly asking speakers to b And And

the way participant

allontana e lo guarda male guarda allontana e lo Krahmer & Swerts (2007) E il gatto grande si il gatto E (EN) intentionally intentionally designed for the

to scare (the birds) birds) (the to scare ;

: Examples of enactments. enactments. Examples of : 9 . . Therefore, asking speakers to gesture would not be a good may may have different (D) ) , suggesting that the gesture encouragement instruction can

are ) s

con gli artigli di fuori a far fuori a paura gli con artigli di claws out claws Figure open | open elicitation elicitation method if the main interest of a study is, for example, assessing how beat gestures specifically impact on prosodic features. that that they may have different “sources” in the speech production process (e.g., beat like movements arising later, at the formulation phase representational gesture phase gesture on example, type Furthermore, that might result in the use enactment. To conclude, these how speakers interpret the prompt to gesture. might interpret the instruction as a request to produce clearer gestures .001 and affect other body articulators and of the in the body storytelling. occurrence of enactments sprezzante

s a to by

ing

dentro dentro

lso, the interact that the

s , Moreover, interest . A

, inside the

spontaneous spontaneous - commented on

suggest in fact

, correction correction and say - can be of gesturers. lexical search. lexical - . This that that they would already participants n and and n omething similar happened S to fetch a fish; she says “arm”

it a a paw, not an arm

do do not gesture much in their

una zampa non un braccio

nside 92 like movements in semi

- ssed in future research, perhaps finding who

with speech productio couple of notable cases that

therefore therefore it puts an arm… interfere quindi quindi mette un brac… data data collected for Study 2, some

: This question may be worth further consideration, as it

EN: (IT the ).

how speakers coped with the instruction to gesture. In one case,

in . instruction instruction can potentially be interpreted in different ways cases, to to cases, h to to a different participant. She described a scene where the cat is on worth discussing uario differently with speech production. depending on the different research questions to be addressed two kinds of instructions can speaker’s lead interpretations to of differences the in prompts the individual that can some To conclude, we believe that both restraining and encouraging the use of gestures can lead to some side effects that can be worth consider l’acq suggests that encouraging speakers to gesture may enhance the speakers’ mental simulation of motor actions while speaking but this can end up, in top top of an aquarium and puts a paw i while enacting the action, and then she makes a self “a paw”: aquarium participant participant described a scene where the cat is holding a mouse in its paw and used the word “hand” instead of “paw”. also different speakers, e.g., those baseline/natural speaking style vs spontaneous high it is explore the encouraging condition prompt by saying gesture a lot without the need to be asked to do it same a way to specifically elicit beat speec Lastly, that this question should be addre ………… … ………… in the into

e … speech and

taking explor

to ed by z in light of previous

in the narration task Conclusions iscussion below

(i.e., (i.e., increased F0 maxima were analy

produced produced

participants participants were asked to describe

stories

, restraining gestures does not seem to 93

vidence that encouraging the use of , acoustic acoustic measures related to pitch and e speech

two two empirical studies aimed the prosodic prosodic features

General D General

. provide s

; by contrast results will be commented In the two studies report

. esults These hypotheses to to a listener and, in doing so, they were either restrained or . The r fluency measures, and speech length and features

c dissertation dissertation ues ues and intensity) is research and terms of val affect speech account account a comprehensive set of measures related to temporal narrative features, intensity. gestures can have effects on prosodi short stories encouraged to gesture. The resulting Th effects of restraining and encouraging the use of gesture on

s s in or the the F

. these mixed

on speech

Indeed,

have an gesture ,

on . These can more words ould his may lead in speech

umber of words speakers cannot n . T

. , using when gestures are restraining “packaging” process

the but but not confirmed in

Based Based on the

Graham & Heywood e expected that . by

ech W

. speech might result to be spe

would would need to integrate different reasons ) and

d any significant difference the visually for in modality modality only narrative speech narrative 2003 evidence that

(

d e expected that speakers not f as as differences in 94

information to be included did on on discourse length, fluency or F0 and utterance length expressed e provide ) spontaneous and and of the - 2 . As a consequence,

detectable on the speech the gesture stream is inhibited impact

On On the other hand,

. ing Finlayson Finlayson et al., speculated speculated that the inability to gesture c come

in semi However, w strategy when be selection

rely .

Kita, Kita, 2000) Restraining the use of gestures of use the Restraining the selection of information to be verbally expressed might

when study study (Chapter measures respectively found longer speech when speakers were unable to n n objects. As a consequence, (Hoetjes et al., 2014; Hostetter et al., 2007)

Speech length 5.1

studies studies did not find any effect of the inability to gesture not significantly

first

ther 1975)

impact on speech chunking variations or speech length o length results, we initially their speech ( shorter. However, ( gesture, or the use of more spatial content phrases and words. Hostetter Hostetter et al., 2007) can be affected speakers, for example, to just exclude some (spatial) information from showing with the hands the shape betwee of an object, or spatial relation information that cannot be (Emmorey & Casey, 2001; Melinger & Levelt, 2004, be related to the example, count on the use of gestures to express some information visually, e.g., As for speech length measures, restrained, speech length would change The does intensity ………… … …………

). The This

. lower … better motor

d to potentially a

condition are trying to

the

(e.g., (e.g., regarding to spatio Appendix).

either lexical access gesture gesture stream leads f speech. This means

day day objects/actions that restraining - - reference that that is expressed verbally n

prosodic planning. prosodic two two experimental conditions no

on differences between the two it might be possible to study study feature actions by animals reduce . . However, the results showed no

prosodic prosodic structure of the speech or more general planning difficulties information 95

, as an effect of or

of possible

through the comparison of . In the former case, it may be that the between the two conditions, whether inhibiting the type motor event

- stories told in the Also,

. and and

spatio assess the

strategies together with the

amount its its semantic richness (as tested but only partly confirmed al., al., 2007)

Rimé et al., 1984) kind of or

the

(Rauscher (Rauscher et al., 1996)

any future, to

(as (as in

Fluency

provide evidence on

comic comic strips that were used in the the

Hostetter et e expected that restraining speakers from gesturing would le aced aced and more disfluent speech nability nability to gesture makes speakers struggle more in finding the right character character trying to put the cat in a cat carrier (see the elements featured in the comic strips were every effects fluency. of the on restraining of use gestures The and people, for example, a cat stealing a cooked chicken from a table, a proposals, gestures can offer a components to powerful be expressed in way the linear to stream o organize that imagistic inhibiting gestures may challenge the speakers who describe (Kita et al., 2017; Kita, 2000) i lexical items when needed. On the other hand, according to the latter W p difficulties utterance utterance length, information structure) of gesture the the impact inability to assess speech information in different prosodic could could be analyzed could speakers to lower the amount of imagery and activity content in their (i.e., the restraining gesture condition vs. the vs. the condition restraining gesture (i.e., In conditions, speech length between the

- A

the has .

s stories

peakers , speakers lowering

free to tell

enhance by

. Also we conjectured lay some positive y were tudies tudies investigating even though spatio and and biomechanically

the

(Marstaller (Marstaller & Burianová, (Bernardis & Gentilucci, by contrast, with no particular challenge rather easy to memorize

However, s has has been found to both controlled by a common

study study that investigated the same .

a were However, along these lines, even

, show that the inability to gesture

96 by the inability to gesture, s Therefore, As a consequence,

and and are

, which

occurring speech -

gesture stream

by drawing upon other cognitive strategies and lexicon.

intensity metrics d the co consistently with or

, (Bernardis & Gentilucci, 2006) of

et al., 2019)

(Hoetjes et al., 2014) short sequences enhancing the compensate

acoustic acoustic analysis, we explored whether restraining gesture Our Our results ts ts participants did with their heads, legs and shoulders had

. Also, Prosodic features , Harrison . for the uld uld be easy to recall by the participants telling the stories 2006; 2006; Krahmer & Swerts, 2007) that restraining gesture use would inhibit prosodic features (Pouw neural network that is connected to Broca’s area 2015) spectral properties how gesture production interacts with speech properties have acoustics suggested that the and two modalities spectral are related both at a motor control level intensity issue directly does not impact F0 and As any effects on acoustic properties such as fundamental frequency and some role in the process of speaking. In other words, the actions imagery to and be described can still be embodied/simulated by other body movements e.g., legs, head, lips and this could still p role in the of process speaking. capitalizing on analytical thinking. though hand gestures were restrained, we cannot exclude that any other movemen simple stories at their own length and pace, imposed on memory motoric thinking was challenged might have consisted consisted of possible explanation of accommodated our for the inability to gesture because results on fluency is that co ………… …

, - - ………… in of to

… impact speech. attempt

be that A possible comfortable

is that, when

might be that (as (as mentioned

in the process

for the absence of prosodic prosodic features of a comparable -

the speakers’ pletely

, well accommodated for

expectations. since the two modalities are

speaking. that encouraging speakers can have

prosodic prosodic features spontaneous posture in some real

. another another explanation could use of gestures of use ger discourse in terms of number of

while A A second explanation , in general,

. showed lon ; however, ) being a the 3 may try to compensate

One can wonder whether, and and expressivity (F0 excursions and intensity).

even ,

). ). ction of modest modest changes in acoustic 4 strategies

their gestures

(Chapter speakers increased speakers’ gesture rate and gesture salience produ for Along these lines,

not be effective

some does not restrain movements com possibly the lack of effect on

higher fundamental frequency maxima and louder louder and higher maxima frequency fundamental ) the 2 the fact that speakers ( ) study study producing hand gestures hand producing

for

1 Encouraging Encouraging day day circumstances (e.g., formal situations), people are used to control our our results did not meet our ( - ), ), and

4 with prosodic effectively eventually

5.2 , namely namely , res nable to gesture, words used; speech The second gesture and led to reduce and reduce and invasive invasive restraining method that was reported to be quite (Chapter life circumstances. some every on speechon as To conclude, the inability to gesture might be due to the fact that we chose a non sitting sitting on hands above and in Chapter speaking, moving other parts of the body u gestu connected at the motor and biomechanical level would speech, prosodic contrasts However, explanation the speech acoustics measures that are typically associated with emphatic

. the not and and and

erent , ously d to can be

gesture suggest Both Both of

speakers s diff evidence . However,

the direct

. predictions

sults for

effect , , , we expected ) might influence

s detrimental 2.4 Our Our re

packaging process 2012 speech content and involving

. However,

these levels possible that

al., gestures gestures and prosodic its

Based Based on the not e expected that gesturing both et

Section Section

( ,

), a focus on may

Cook 1.3

impact on it is also at some point

above

see , with can 98 gesture the idea that

required for speech production

disentangle disentangle whether these

information to be included in speech es

to analysis planned planned in the process of speech production - A possible explanation of our results is that, the

. (1) to speak more fluently. explore

at least two levels attention attention to producing gesture models (Section

the major theoretical models for speech

attention to

textual on

Along Along these lines of prosodic prosodic structuring

has has been discussed speakers to speak more fluently

(2) explored. we did not find evidence that having speakers produce

urther speakers’ f may may result in different speech chunking strategies drawing

what

packaging process and

in light

the selection of are potentially co Hufnagel, 2019) Hufnagel, to

- memory). that , , will be needed

Fluency Speech Speech length previously drawing encouraging the use of gesture showing working when encouraged to gesture, speakers thinking might have about been producing consci more Therefore, the prompt to gesture might have increased the demand on the gestures while telling speakers’ the working memory (though, stories more would lead influence of encouraging gestures on fluent (connected) speech ha been As for fluency, more gestures leads them of the main theoretical structures (Shattuck in the future, structure explained production that that both the selection potentially involve of information and that processes: information these levels different amount of information included in speech. Also, it may well be Similarly Similarly that speech production ………… …

I ………… . that s are … 1.3 speech

this would the . . This would while telling investigation he result t have found in

is imaging studies

h - result in changes visual visual data (i.e., However, we did

in Section .

and

2019) over time claimed to play a role , it may well be (

believe that

measures, of different strength and

is detrimental to speech fluency

, it may well be that asking s reviewed interfered with et al.

acoustics and and have

, gesture Dalla Dalla Volta, 2008; Marstaller &

studies studies

control system the gestures to produce 99

generally generally speech speech speech rate analysis analysis of our audio

biomechanic interdependence between

, Harrison, a a more direct “enhancing effect” on the on plan for

(F0) and intensity whether

s would would enhance speech

speech synchrony.

- ha a motor Pouw urther

Gentilucci & impact had to potential may have

explore that (Bernardis & Gentilucci, 2006; Krahmer & Swerts, most most important finding reported in t

atures Finally, Finally, as

. e.g.,

intensity intensity (as also found in Hoetjes et al., 2014) predictions of the (

. restraining the use of gesture does not seem to impact

) can set and and this

can can directly in the future,

of speech, and thus not cause them to speak faster at all. speak them faster at to cause not thus of speech, and

, Probably the Prosodic fe . telling - roca’s area clarity steady steady vocalizations produced in the lab in the emergence of gesture kinematics serve to explore the speech and gestures Burianová, 2015) Burianová, However story claiming claiming that speech and gesture production are both common controlled motor control system by engaged with a neural a network connected to B was that, while speech F0 and encouraging gestures’ use acoustics of speech. This idea is also supported by brain enhancing the gesture stream in acoustic patterns 2007) As As for fundamental frequency in line with the speech and gesture share fluency in any way speakers to gesture more while speaking can have led them to rather aim for the stories planning and production processes not favoring not find that instructing speakers to gesture encouraged to gesture

the and and

Also,

on on would modality. different effects

ly spontaneous narrative - prosodic prosodic contrasts role of gestures on together together with a biomechanical potential

: . ch. ch. At the same time, it showed that 100 the on on prosodic features (as suggested by

s and and

direct mechanisms and functions that gestures . . Further investigations in this directi the

Hufnagel, 2019) Hufnagel, - ing gesture gesture production production

clarify and and prosodic structure are jointly planned and produced. produced. and planned are jointly structure prosodic (Shattuck

gesture and and gesture prosodic prosodic and acoustic features of spee the inability to gesture can be well accommodated speech in production terms of and fluent spoken prosody in semi speech. However, future work is needed to further explore how (and if) To conclude, the main contribution of this dissertation is that it explored the potential functions of gesture production in the speech planning and articulation phase, suggesting that gesture production could enhance have for speech planning and generation of prosodic structures in the process of speech production a a future contribution would be to explore the potential of different types of gesture Krahmer & Swerts, 2007) contribute to articulation articulation interdependence between the two modalities, expressivity can be enhanced and “helped” by the gesture shed more light on the complex interconnectedness between speech ………… …

. ………… (2), (2), Why (11), (11), John … 14 9 , ,

Proceedings Proceedings of the 9th References 665. 665. Revista Do GEL – , 661 , The The gestural origin of language Amsterdam/Philadelphia: Meadow, Meadow, S. (2015). Gesture for

Prosody

101 Linguistics and Language Compass 38). 38). – term speaking rate. - rd: Oxford University Press. Oxford University rd: How the brain got language: The mirror system (pp. 15

(Vol. 16). Oxfo 16). (Vol.

unicating 197. 197. 451. – – New Press. York: Oxford University 183 hypothesis for the estimation of long Speech on Conference International Minimum Sample Length for Speaking Rate. comm Benjamins Publishing Company Representational gestures help speakers speaking. package In R. B. information Church, M. W. for Alibali, & S. D. Kelly Gesture?: (Eds.), How the hands function in speaking, thinking and Linguists: Linguists: A Handy Primer. 437 libali, M. W., Yeo, A., Hostetter, A. B., & Kita, S. (2017). Armstrong, Armstrong, D. F., & Wilcox, S. E. (2007). Arbib, M. A. (2012). Arantes, Arantes, P., & Lima, V. G. (2017). Towards a Methodology To Estimate Arantes, Arantes, P., Eriksson, A., & Lima, V. G. (2018). Minimum sample length A Abner, N., Cooperrider, K., & Goldin

, – – ). ). 9

. tongue Journal Journal - (1), (1), 289 (4), (4), 394 Journal Journal of 1610. 1610. the - – 57 21 of , , - )

Acta Psychologica Acta 56. – (11), 1605 (11), ilitates the Learning of , 35 , 21

, logy Bulletin 90 , Studies Studies in Second Language 489. –

Fuente, Fuente, S., & Prieto, P. (201 -

520.

ArXiv Preprint ArXiv:1406.5823 Preprint ArXiv – of dialogue and visibility. (4), 469 (4),

17. 102 – 15 ,

(2), 495 (2), . (1), 1 (1), Psychological Science 2 58 58 , , – Meadow, Meadow, S. (2010). Gesture Changes Thought by - 33 , lan, lan, J. (1999). An experimental investigation of the Gesture Personality Personality and Social Psycho British Journal of Psychology of Journal British González, González, N., González activation on activation L1 and L2 speech fluency. - -

41(1) , g It in Action. in It Action. g Discourse Processes 35. – effects models using lme4. effects using models -

, 25 , language co 161 of of the Royal Statistical Society: Series B (Methodological 300. Groundin Rate: A Practical and Powerful Approach to Multiple Testing. role of iconic gestures in lexical access using phenomenon. the tip experimental study of when communicate. and how speakers use gestures to telephone: Independent effects Language and Memory 405. gestures. mixed for Dialogue. Observing and Producing Pitch Gestures Fac Mandarin Chinese Tones and Words. Acquisition

Bergmann, Bergmann, C., Sprenger, S. A., & Schmid, M. S. (2015). The impact of Benjamini, Y., & Hochberg, Y. (1995). Controlling the False Discovery Beattie, Beattie, G., & Cough Beilock, S. L., & Goldin Bavelas, Bavelas, J., Kenwood, C., Johnson, T., & Phillips, B. (2002). An Bavelas, Bavelas, J., Gerwing, J., Sutton, C., & Prevost, D. (2008). Gesturing on the Bavelas, Bavelas, J., Chovil, N., Lawrie, D. A., & Wade, A. (1992). Interactive Bates, Bates, D., Mächler, M., Bolker, B., & Walker, S. (2014). Fitting linear Bavelas, J., Chovil, N., Coates, L., & Roe, L. (1995). Gestures Specialized Baills, F., Suárez ………… … , , – – ………… … (2), (2), 257 (4), (4), 539 Journal Journal of

42 , 136 , 190.

– Amsterdam, Amsterdam, The 90.

Meadow, Meadow, S. (2007). Tongue Tongue Experience. – speech gestures are - - - the - (2), 178 (2), 2), 79 2), of – - 44 , (Vol. (Vol. 70) (1

, 223. es

– 300.

– thought thought and co 103 -

. Sex Rol Sex Journal Journal of Experimental Psychology: General (2), 204 (2), Neuropsychologia eview of the Tip (2), 285 (2),

3 109 , , The The Journal of the Acoustical Society of America

Cerebral Control of Speech and Limb Movements.

. 2658. Advances Advances in Psychology – on system. system. on 116. 116. – Journal Journal of Experimental Psychology: General & Kita, S. (2016). Co ), 2649 ),

0). 0). In, (1), 102 (1), (5 generated by the Experimental Psychology: Learning, same Memory, and Cognition action generation process. spatial problem solving. 140 (199 Netherlands: Elsevier Science of the role of intentionality in gesture production and comprehension. Saperi,Reti, Linguaggi 550. 550. Psychological Bulletin Making Children Gesture Brings Out Implicit Knowledge and Leads to Learning. http://www.praat.org/ communication. nonverbal vocal fry registers. 103 [Computer program]. Version 6.0.43. Retrieved from communicati aerodynamic, physiologic, and perceptual properties of modal and Chu, Chu, M., Chu, Chu, M., & Kita, S. (2011). The nature of gestures’ beneficial role in Campisi, E., & Mazzone, M. (2016). Do people intend to gesture? A review G. R. Hammond (Ed.) Brown, Brown, A. S. (1991). A R Broaders, Broaders, S. C., Cook, S. W., Mitchell, Z., & Goldin Briton, Briton, N. J., & Hall, J. A. (1995). Beliefs about female and male Boersma, Boersma, P., & Weenink, D. (2018). Praat: doing phonetics by computer Blomgren, Blomgren, M., Chen, Y., Ng, M. L., & Gilbert, H. R. (1998). Acoustic, Bernardis, P., & Gentilucci, M. (2006). Speech and gesture share the same , :

ist Up

- (pp.

(02), (02), 475.

– 26 (Vol. (Vol. 7). 610.

fferences fferences , – d thought Why Why gesture? (4), 465 (4), stures, but not in Gesture and

(4), 594 (4), 63 Proceedings Proceedings of the American American Scient ,

27 ,

16. 16. – accompanying gestures: the role - Journal Journal of Experimental Psychology: Meadow, Meadow, S. (2018). The Palm

(23), 1 (23), - 3 Behavioral Behavioral and Brain Sciences , Meadow, Meadow, S. (2010). Gesturing makes Meadow, Meadow, S. (2012). Ge 104 - - John Benjamins PublishingJohn Company.

Cambridge Press. University 709. – Journal of Memory and Language and Memory of Journal handedness. handedness. - Language and Cognitive Processes Cognitive and Language The The Cambridge handbook of metaphor an ask ask Increases Speakers’ Gesture Rate, Gesture Salience

(2), 694 (2), Cambridge:

145. 143 – , Frontiers in Communication Frontiers in 208. 208. 501). 501). the the hands function in speaking, thinking and communicating

– – , 138 , a Narration T and the Production of Representational Gestures. evolution of right 199 today today as a “behavioral fossil” coupled to speech. 87 Puzzle: Puzzle: Meanings and Origins of a Widespread Form Sign. language may have evolved from manual gestures, which survive meaningless meaningless movements, lighten math. explaining working memory load when memories that last. memories last. that Gibbs JR (Ed.), 483 General how Amsterdam/Philadelphia: in frequency and saliency of speech of cognitive abilities and empathy. 270.

Cravotta, Cravotta, A., Prieto, P., & Busà, M. G. (2019). Encouraging Gesture Use in Corballis, Corballis, M. C. (2003). From mouth to hand: Gesture, speech, and the Corballis, Corballis, M. C. (1999). The Gestural Origins of Language: Human Cooperrider, K., Abner, N., & Goldin Cook, Cook, S. W., Yip, T. K., & Goldin Cook, S. W., Yip, T. K., & Goldin Cienki, A., & Müller, C. (2008). Metaphor, gesture, and thought. In R. W. Church, Church, R. B., Alibali, M. W., & Kelly, S. D. (Eds.). (2017). Chu, Chu, M., Meyer, A., Foulkes, L., & Kita, S. (2014). Individual di ………… …

: . , . , ………… (6), (6), … ishing 141 (Vol. (Vol. 7,

, 98. Cambridge: – rica Iazykovedenie Iazykovedenie . Why Why Gesture?: 311) – (1), 49 (1), : : King’s Crown Press. 1 , UK

Personality Personality and Individual Behavior Behavior Research Methods John John Benjamins Publ

Semiotica Oxford,

(Vol. (Vol. 2, pp. 284

antichi investigata nel gestire napoletano gestire nel investigata antichi 105 . . Gesture, thought and spatial language.

. esture 26. G - 365.

– Comes, Comes, J., Asor, E., Swerts, M., & Prieto, P.

- , 21 , 173. – 50. 50. –

La mimica degli degli mimica La (2), 355 (2), Amsterdam/Philadelphia: Gesture and environment. , 105 , The The Journal of the Acoustical Society of Ame 390. 28 m – , Language Language and

(1), 35 (1), P. P. (2017). The asymmetric redundancy of gesture and speech. 75). 75). 1

M., M., & Furnham, A. (2000). Personality and speech production – , - (Ed.), (Ed.), Gibert, N., Borràs (2), 385 (2),

- (2017). (2017). The timing of head movements: The role of prosodic heads and edges. Gesture Categories, origins, usage, and coding. coding. and usage, origins, Categories, [Observations [Observations on reflexes and issues in language study]. i Materializ Cambridge Press University A pilot study of second language learners. Differences Company. D. In R. B. Church, M. W. Alibali, & S. D. Kelly How the hands function in speaking, thinking and communicating (Eds.), pp. 59 and and measure speech rate automatically. 41 Paderborn, Germany del stamperia e Fibreno.Napoli: cartiera 6th 6th Conference on Gesture and Speech in Interaction (GeSpIn) Emmorey, K., & Casey, S. (2001) Esteve Efron, Efron, D. (1941). Ekman, P., & Friesen, W. V. (1969). The repertoire of nonverbal behavior: Dobrogaev, Dobrogaev, S. M. (1929). Ucnenie o reflekse v problemakh iazykovedeniia Dewaele, J. De Ruiter, J. P. (2000). The production of gesture and speech. In McNeill de Ruiter, J. De Jong, N. H., & Wempe, T. (2009). Praat script to detect syllable nuclei De De Iorio, A. (1832).

– The 24). 24).

– 864. (2), (2), 107 Individual Individual – Journal Journal of

9 etherlands: etherlands: , . . New York: 215. 215. – (3), 850 (3),

56 , (Vol. (Vol. 90, pp. 21 357. (2), 193 (2),

–

34 , 62. – Language Language Pathology , , P. A. (2000). Hand Gestures: - . . Amsterdam, The N (4), 353 (4), 8 58)

, – (1), 43 (1), 3 106 , 2014). 2014). Elaborating who’s what: A study of tures tures on disfluency. In R. Eklund (Ed.),

Language and Hearing Research and Language

(Vol. (Vol. 70, pp. 29 , Joint Joint Arm Movements. In G. E. Stelmach & P. A.

. - Language Pathology Language Advances Advances in Psychology. Cerebral Control of Speech and -

American American Journal of Speech y y and rate in ALS dysarthria: A case study. Australian Journal of Linguistics of Journal Australian 4739. 4739.

– Gibert, N., & Prieto, P. (2013). Prosodic Structure Shapes the - Horbury, Horbury, D., & Guttentag, R. E. (1998). The effects of restricting - intelligibilit Medical Speech Perspectives and Preliminary Implications for Adults with Acquired Dysarthria. 115. hand hand gesture production on lexical retrieval Psychology of Journal American and free recall. Limb Movements Elsevier Science Göteborg, Sweden.Göteborg, Jaw and Multi Vroon (Eds.), restriction restriction of hand ges Proceedings of Disfluency in Spontaneous Speech Workshop (DiSS ’03), Gothenburg Papers in Theoretical Linguistics differences differences in language ability Press. Academic and language behavior constructed constructed action and clause structure in Auslan language). (Australian sign Temporal Realization of Intonation and Manual Gesture Movements. Speech of Journal 4727

Garcia, Garcia, J. M., & Cobb, D. (2000). The effects of gesturing on speech Garcia, Garcia, J. M., Cannito, M. P., & Dagenais Frick Flanagan, Flanagan, J. R., Ostry, D. J., & Feldman, A. G. (1990). Control of Human Fillmore, C. J., Kempler, D., & Wang, W. S. Y. (Eds.). (2014). Finlayson, S., Forrest, V., Lickley, R., & Beck, J. M. (2003). Effects of the Ferrara, Ferrara, L., & Johnston, T. ( Esteve ………… … , ………… The (1), (1), … 64 , European European

585. 585. Journal Journal of –

957. – 1293. (5), 573 (5),

– 8 , Psychological Science (6), 944 (6), 180. 180. – 61 ,

(6), 1282 (6), 41 195. speech gesture: Evidence from , – -

(2), 174 (2), Annual Annual Review of Psychology

132

162. 162. , 107 – (2), 189 (2), on (Vol. (Vol. 53). Amsterdam, The Netherlands: 5

Cogniti (2), 153 (2), al al codability on speech performance. s s in mental rotation: What does gesture tell us? 14 , 195. –

Meadow, Meadow, S., Newcombe, N. S., & Shipley, T. (2013). , , Nusbaum, H., Kelly, S. D., & Wagner, S. (2001). - Perspectives Psychological Science on 522. –

(2), 157 (2), 4 Language and Hearing Research Hearing and Language , , & Bavelas, J. (2004). Linguistic influences on gesture’s form. ,

, 283. 283. – Meadow, Meadow, S., & Alibali, M. W. (2013). Gesture’s Role in Speaking, Meadow, S. (6), 516 (6), eech - - Journal of Social Psychology of Journal John Benjamins Publishing.John gestures and of verb 12 Studies in Corpus Linguistics Learning, Learning, and Creating Language. 257 Explaining math: gesturing lightens the load. Individual Individual difference Processing Cognitive Embodiment. Verbal working memory predicts co differences.individual gestures gestures are controlled by the Psychology Experimental of Quarterly Journal same motor control system. Gesture contribution contribution of iconic gestures and message predictiveness. Sp Graziano, Graziano, M. (2014). The development of two pragmatic gestures of the Graham, Graham, J. A., & Heywood, S. (1975). The effects of elimination of hand Götz, Götz, S. (2013). Fluency in native and nonnative English speech. In Goldin Goldin Glenberg, A. M., Witt, J. K., & Metcalfe, J. (2013). From the Revolution to Göksun, T., Goldin Gillespie, M., James, A. N., Federmeier, K. D., & Watson, D. G. (2014). Gerwing, J. Gentilucci, Gentilucci, M., & Dalla Volta, R. (2008). Spoken language and arm Garcia, J. M., & Dagenais, P. A. (1998). Dysarthric sentence intelligibility:

–

179. 179. –

514. (pp. 311 289. 289.

–

– 267. –

(2), 149 (2), 362. 362. 8 – , , 257 , (3), 495 (3), 604. 604. (2), 267 (2), –

57 15 24

, , 347 , , , d the Gestural Origin of 172. Gesture 37 – 84. 84. , – (3), 600 (3), gestures, gestures, imagery, and word

11 From Gesture in Conversation to ,

2), 147 2), ,

– nd nd Crosslinguistic Comparisons. (S1), 65 (S1),

(1 17. 17. 33 – 591. , – 108 115 , Speech Communication (879), 1 (879), 9 (4), 575 (4), , 33 Semiotica , Psychonomic Bulletin & Review Bulletin Psychonomic Language and Cognitive Processes Cognitive and Language Annual Review Neuroscience of Annual System hmer, E., & Swerts, M. (2014). Does our speech change Current Anthropology Current called called Open Hand Supine family in Italian children. In M. - Journal of Speech Hearing Research of and Journal action. simulated mutually mutually shared knowledge influences task. narrative speech and gesture in a mechanism. when gesture? we cannot Language. presentation speech: A proposal for an automatic feedback Single Words. issues in the study of language development. development. of language inissues the study retrieval in speech. Evidence From Developmental a Psychology Frontiers in Seyfeddinipur & M. Gullberg (Eds.) Visible Action as Utterance: Essays in Honor of Adam Kendon. Benjamins Publishing John Company. Amsterdam: 330). so Hostetter, Hostetter, A. B., & Alibali, M. W. (2019). Gesture as simulated action: Hollien, H., & Michel, J. F. (1968). Vocal Fry as a Phonational Register. Hostetter, A. B., & Alibali, M. W. (2008). Visible embodiment: gestures as Holler, J., & Wilkin, K. (2009). Communicating common ground: How Hoetjes, M., Kra Hincks, R. (2005). Measures and perceptions of liveliness in student oral Hagoort, Hagoort, P., & Indefrey, P. (2014). The Neurobiology of Language Beyond Hewes, G. W. (1992). Primate Communication an Hadar, Hadar, U., & Butterworth, B. (1997). Iconic Graziano, Graziano, M., & Gullberg, M. (2018). When Speech Stops, Gesture Stops: Gullberg, M., De Bot, K., & Volterra, V. (2008). Gestures and some key ………… …

, , ,

– – ………… (1), (1), … 82 , (1), (1), 62 nic nic Hand (3), (3), 721 12 26 , ent of polite , Language Language and Nashville, Nashville, USA.

, . : : How prosody and

Gesture

1102 - Frontiers in Psychology 862. 862. – 1097 Brain Brain and Language Universitat Universitat Pompeu Fabra , (2007). (2007). Does sitting on your Proceedings Proceedings of the 29th Annual 29 , Journal Journal of Consciousness Studies Journal Journal of Speech

(05), 825 (05),

46 109 PhD Diss: , gesture gesture production.

. Psychonomic Psychonomic Bulletin & Review 1012. – , 228. , Meadow, Meadow, S. (1998). Why people gesture when - , 996 , 396 , 48 , Preschoolers ’ pragmatic development

Nature

16. 16. . –

Journal of Child Language Child of Journal

29. – (255), 1 (255), on, on, J. M., & Goldin emergence of speech and gesture. 2 they speak. Hearing Research production components: A critical update. Supplementation Supplementation Strategies: Effect of Alphabet Cues and Ico Gestures on Dysarthric Speech. Barcelona stance in preschoolers: how prosody, gesture, and body cues pave the way. 83. 83. gesture lend a helping hand 22 situation on representational on on speech during motor descriptions. Meeting of the Cognitive Science Society on the production of lexical movements. 752. hands make you bite your tongue? The effects of gesture prohibition Revisiting Revisiting the framework. Iverson, Iverson, J. M., & Thelen, E. (1999). Hand, mouth and brain. The dynamic Ivers Indefrey, P. (2011). The spatial and temporal signatures of word Hustad, Hustad, K. C., & Garcia, J. M. (2005). Aided and Unaided Speech Hübscher, I., Garufi, M., & Prieto, P. (2019). The developm Hübscher, I. (2018). Hostetter, Hostetter, A. B., & Potthoff, A. L. (2012). Effects of personality and social Hostetter, Hostetter, A. B., & Hopkins, W. D. (2002). The effect of thought structure Hostetter, Hostetter, A. B., Alibali, M. W., & Kita, S.

. ,

– – – 7 , (2), (2), 56 , (2), (2), 157 (1), (1), 109 (3), (3), 416

Cambridge: 16 26 16

, ,

. , 108. 108. – ure Gesture Gest (1), 92 (1), 2 , Language, Language, Cognition and The The Relationship of Verbal and

The Mouton Hague:

Studies Studies in Dyadic Communication

. 227. 110 – 366. Journal Journal of Memory and Language –

229 1 : : visible action as utterance

Annual Annual Review of Anthropology – , 207 , 214 (3), 348 (3), 1 : Pergamon Press: 8 Gesture Pointing: Pointing: Where language, culture, and cognition meet , nication

In M. R., Key (Ed), 40. 34(9), 34(9), –

04). 04). , Journal of Linguistic Anthropology of Journal Gesture

New York

12), 19 12), 303. 303. – 210,

N., N., & Garnham, A. (2007). The role of conversational hand - –

(11 Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc., Publishers. Inc., Mahwah, New Associates, Lawrence Erlbaum Jersey: and ‘sign.’ ‘sign.’ and 175. Cambridge Press. University (Emblems). 128. of utterance. Commu Nonverbal In Siegman, A. & Pope, B. (Eds), 177 mechanisms of Neuroscience speech production. restriction restriction on quality of narrative production. 431. gestures gestures in a narrative task. 291 6 Kita, Kita, S. (Ed.). (2003). Kendon, Kendon, A. (2017). Pragmatic functions of gestures. Kendon, Kendon, A. (20 Kendon, A. (2008). Some reflections on the relationship between ‘gesture’ Kendon, Kendon, A. (1997). Gesture. Kendon, Kendon, A. (1992). Some Recent Work from Italy on Quotable Gestures Kendon, Kendon, A. (1980). Gesticulation and speech: Two aspects of the process Kearney, E., & Guenther, F. H. (2019). Articulating: the neural Kendon, A. (1972). Some relationships between body motion and speech. Jenkins, T., Coppola, M., & Coelho, C. (2017). Effects of gesture Jacobs, ………… … ,

– ………… … . New

(3), (3), 1

Journal Journal of

Cambridge 8(1) , 414. Language Language and – 167. 167. – (Ed.), (Ed.),

(3), 396 (3), linguistic variation in Cambridge: accompanying accompanying gesture: - - (2), 145 (2), 57 , D. , 24 ,

185). 185). conceptualization conceptualization hypothesis. – - Gesture and Sign Language in for Laboratory Phonology

- (1997). (1997). Movement phases in signs

: Cambridge: Press. University

32.

– (pp. 162 111

35). 35). – (1), 16 (1), Cognitive Processes Cognitive p. 23 p.

Cambridge 48 (124), 245. (124), Memory and Language and Memory , 3 , cultural cultural variation of speech - 283). 283).

– Speech production and second language acquisition

Journal of of Journal Language and Language (Vol. 1371, p 1371, (Vol.

Language Language and Gesture (pp. 261 Computer Computer Interaction. Proceedings of the International Gesture speech gestures, and their transcription by human coders. In I.

- - , , J., Tiede, M. K., & Tyrone, M. E. (2017). A Kinematic Study of ć Alibali, Alibali, M. W., & Chu, M. (2017). How do gestures influence

(2000). (2000). How representational gestures help speaking. In McNeill van van Gijn, I., & van der Hulst, H.

, , & Özyürek, A. (2003). What does cross (Ed.), (Ed.),

. .,

. . 26. Gesture Prosodic Structure in Articulatory and Manual Gestures: Results from a Novel Method of Data Collection. perception. lexical access: a process model. In McNeill York: Routledge. Taylor & Francis Group. & Francis Taylor York: Routledge. prominence: Acoustic analyses, auditory perception and visual semantic semantic coordination of speech and gesture reveal?: Evidence for an interface representation of spatial thinking and speaking. Language and Memory D. Press.University Human Workshop Psychological Review and co Wachsmuth & M. Fröhlich (Eds.), A review. A thinking and speaking? The gesture Krivokapi Krauss, Krauss, R. M., Chen, Y., & Gottesman, R. F. (2000). Lexical gestures and Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic Kormos, Kormos, J. (2014). Kita, S Kita, S Kita, S Kita, S Kita, S. (2009). Cross , , -

284 Coh , 89. – 502. 502. –

Journal Journal of Blackwell.

. . Cambridge, - (1), 71 (1), , 498 , 3 , Neuroscience

The The neurocognition of . . Retrieved from age: a blueprint of the

Journal Journal of Psycholinguistic 304. – Psychonomic Bulletin & Review stures. ysis: An introduction to fieldwork Oxford University Press. Oxford University

Laboratory Phonology (2), 299 (2), L2 L2 pronunciation in a reading aloud 112 10 , . Quebec City, Canada. Quebec City, . . Malden, Massachusetts: Wiley Massachusetts: Malden, .

, Oxford, UK: Oxford,

Proceedings Proceedings of the 36th Annual Meeting of the Speaking: Speaking: From intention to articulation 89. 89. – Phonetic Phonetic data anal 122).

– chuk, chuk, H., & Casasanto, D. (2014). Beat gestures facilitate (1), 69 (1), 500. 500. – fect on working memory. 27 (pp. 83 ,

9th International Conference on Speech 2018 Prosody on Conference International 9th 41. – (3), 496 (3), Coromina, Coromina, J., Prieto, P., & Rohrer, P. (2018). Brief training with - http://cohmetrix.com/ Speech Hearing Research and Metrix version 3.0 [Computer software] Research 20 network for overt production of speech and gesture. 29 Cognitive Science Society Cognitive gesture ef between intonation and gesture. gesture. between and intonation speech production. rhythmic beat gestures helps task. speaker. In P. Hagoort & C. . M. Brown (Eds.), language and instrumental techniques instrumental and MIT press. Massachusetts:

McGlone, McGlone, R. E. (1967). Air flow during vocal fry phonation. McNamara, D. S., Louwerse, M. M., Cai, Z., & Graesser, A. (2013). McClave, McClave, E. (1998). Pitch and Manual Ge Marstaller, Marstaller, L., & Burianová, H. (2015). A common functional neural Marstaller, Marstaller, L., & Burianová, H. (2013). Individual differences in the Lucero, Lucero, C., Zahar Llanes Loehr, D. P. (2012). Temporal, structural, and pragmatic synchrony Levelt, Levelt, W. J. M. (1999). Producing spoken langu Levelt, Levelt, W. J. M. (1989). Ladefoged, P. (2003). ………… … , . e – …………

(3), (3), … 24 (Vol. (Vol. 7, ,

speaking. (3), (3), 544 - 14 for - , Why Why Gesture?: 161). 161). Cambridge: Journal Journal of Speech,

–

. Milano: Muggiani. Milano: . pare? 1236. 1236. – 141. (pp. 141 –

John John Benjamins Publishing

. . Chicago: University of Chicago 664. 664. thinking thinking and communicating (2), 119 (2), 1224 (4),

– Koenderman, M. (2013). Gesturing

- 4

Psychonomic Bulletin & Review The The American Journal of Psychology 56 , , 113 telling telling style: evidence from bilinguals. -

al dizionario italiano al dizionario

(4), 641 (4), Gesture speech unity. What it is, where it came from. 10 - , Meadow, Meadow, S. (2017). Gesture as representational Language Language and Gesture -

Hand Hand and mind: What gestures reveal about thought Gesture and thought Journal Journal of Speech and Hearing Research (Ed.), (Ed.), Supplemento Amsterdam/Philadelphia:

424. – , D.

& Duncan, S. (2000). Growth points in thinking 102).

– ., ridge Press. University

(3), 411 (3), action: A paper about function. frequency frequency is linked to story Cognition and Language Fry Fry Phonation. 551. working memory and speech. 117 by Speakers With Aphasia: How Does It Com Hearing Research and Language, Camb of theintention speaker. Company In McNeill In R. B. Church, M. W. Alibali, & S. D. Kelly How the hands function in speaking, (Eds.), pp. 77 Chicago: University of Chicago press. of Chicago University Chicago: Press. Novack, Novack, M. A., & Goldin Nicoladis, Nicoladis, E., Nagpal, J., Marentette, P., & Hauer, B. (2018). Gestur Murry, Murry, T. (1971). Subglottal Pressure and Airflow Measures During Vocal Morsella, Morsella, E., & Krauss, R. M. (2004). The role of gestures in spatial (1963). B. Munari, Mol, Mol, L., Krahmer, E., & van de Sandt Melinger, A., & Levelt, W. J. M. (2004). Gesture and the communicative McNeill, D McNeill, D. (2017). Gesture McNeill, D. (2005). McNeill, D. (1992). . – (6), (6), 10 ,

(4), (4), 602

34 ,

B113. B113. – 371. 8. 8. – – 132. 132. – , 1 ,

69 , (3), 357 (3), (3), B101 (3), n on spatial language in (2), 97 (2), 15 96 , , Cognitive Cognitive Science 77 Developmental Developmental Science ,

London, UK. London,

Gesture

Cognition 114 1476 Cognition - esent objects. Proceedings Proceedings of the 39th Annual Meeting of the Comunicare Comunicare Senza Parole. La Comunicazione 1471 Speech Communication ,

Meadow, Meadow, S. (2005). Gesture is at the cutting edge -

98. – Meadow, Meadow, S. (2010). Gesturing saves cognitive resources ‐ 238. – (1), 71 (1),

., ., & Goldin 2 Ş , 219 , , ni 754. 665. Verbale Nel Bambino e Nell’interazione Sociale Tra Adulti. Roma:

- – – kan, 2 ş of speech models. monitoring Bulzo Gesture emblematico olofrastico. Non 747 when talking about nonpr 619. Retrieved from http://www.liwc.net/ on children’s lexical retrieval ability. gesture moregesture when to? instructed Linguistic Inquiry and Word Count: LIWC 2015 [Computer software] T. T. (2017). The effects of gesture restrictio young and elderly adults. Science Society Cognitive of early language development. development. language of early extroversion extroversion on communication: Evidence visibility manipulation. from an interlocutor 65 Postma, Postma, A. (2000). Detection of errors during speech production: a review Poggi, Poggi, I. (2002). Symbolic gestures: The case of the Italian gestionary. Poggi, Poggi, I. (1983). La mano a borsa: analisi semantica di un gesto Ping, Ping, R., & Goldin Pine, K. J., Bird, H., & Kirk, E. (2007). The effects of prohibiting gestures Pennebaker, J. W., Booth, R. J., Boyd, R. L., & Francis, M. E. (2015). Parrill, Parrill, F., Cabot, J., Kent, H., Chen, K., & Payneau, A. (2016). Do people Özçali Özer, D., Tansan, M., Özer, E. E., Malykhina, K., Chatterjee, A., & Göksun, O’Carroll, O’Carroll, S., Nicoladis, E., & Smithson, L. (2015). The effect of ………… … . – n …………

Upper … , , 201

speech 325. – Cognitive – 615. 615. – Giménez, I. - rgh, Scotland, Proceedings Proceedings of speech physics: – (4), 311 (4),

8 (3), 610 (3), , 10 14. 14. , – , 1 , 33. 33. Edinbu 5 – , movements, especially for PsyArxXiv, PsyArxXiv, May 2019, , , 29

231. 115 – Voicing Motivation and Emotion and Motivation

Monitoring Monitoring of Speech. -

(4), 226 (4), 277. 277. International International Conference on Speech Prosody – ment and lexical access: Do noniconic gestures 7 Frontiers in Psychology in Frontiers , Social Resonance: Acoustic Information about Psychonomic Bulletin and Review and Bulletin Psychonomic er, J. A., van Gog, T., Zwaan, R. A., & Paas, F. (2014). Journal Journal of Experimental Psychology: General (3), 269 (3), 17 , mechanical basis for the emergence of gesture

Error Detection in DiSS’01:Disfluency in Spontaneous Speech Self relative immobilization on the speaker’s nonverbal behavior and o imagerythe level. dialogue aid in retrieval? 205. 205. lexical access: The role of lexical movements in speech production. Psychological Science (2018). (2018). Deconstructing Proceedings beat of the 9th gestures: A labelling proposal. Vick, W. (2019). Limb Movement https//doi.org/10.31234/osf.io/ny39e in mental mental problem solving reduces individuals eye with lower visual working memory capacity. Processing The bio synchrony. http://dx.doi.org/10.1037/xge0000646 Toward a more embedded/extended perspective on of gestures. function the cognitive Rimé, B., Schiaratura, L., Hupet, M., & Ghysselinckx, A. (1984). Effects of Seyfeddinipur, M., & Kita, S. (2001). Gesture as an Indicator of Early Ravizza, Ravizza, S. (2003). Move Rauscher, Rauscher, F. H., Krauss, R. M., & Chen, Y. (1996). Gesture, speech, and Prieto, P., Cravotta, A., Kushch, O., Rohrer, P. L., & Vilà Pouw, Pouw, W., Paxton, A., Harrison, S., Dixon, J. A., Grossma, S., Hart, O., & Pouw, Pouw, W., Mavilidi, M.F., van Gog, T., & F. Paas, (2016). Gesturing during Pouw, Pouw, W., Harrison, S. J., & Dixon, J. A. (2019). Gesture Pouw, W., de Nooij . - - -

sive (pp.

267. 267. speech - lecture – -

. . Cambridge,

(3), 239 (3), 105. –

27 , 360. 360. – , 83 ,

13. 13. – Language, Language, Cognition and n: a comparative analysis. (3), 327 (3),

Semiotica 12 (1514), 1 (1514),

, 9

654. 654. 842. – 116 rieto, P. (2019). Observing storytellers Gesture – rsity Press.rsity national national Congress of Phonetic Sciences

h gestures in a sample of academic (4), 629 (4), (3), 837 (3), Origins of human communication 30 Language Language in hand: Why sign came before speech

, speec 108 - , 12. 12. – Frontiers in Psychology Frontiers in , 1 , Research on Language and Social Interaction and Research Language on Cognition 1494). 1494). Melbourne, Australia: Canberra, Australia: Australasian – Hufnagel, Hufnagel, S., & Ren, A. (2018). The prosodic characteristics of Hufnagel, Hufnagel, S. (2019). Toward an (even) more comprehen Hufnagel, S., & Prieto, P. (2019). Dimensionalizing co referential co - - - -

Giménez, I., Igualada, A., & P who use rhythmic beat gestures improves children’s narrative Massachusetts: MIT press. Massachusetts: information status in Dutch and Phonetics of Journal Italia Washington: Gallaudet Unive Gallaudet Washington: Author. style speech. style Speech Science and Technology Association Inc. Association Speech Technology Science and non gestures. gestures. In S. Calhoun, P. Escudero, M. Tabain, & P. Warren (Eds.), Proceedings of the 19th Inter 1490 model of speech production Neuroscience planning. themselves in managing problems in speaking: Evidence from self repairs. UK. - Vilà Tomasello, Tomasello, M. (2008). The of gesture. K. (1993). Tuite, production Swerts, M., Krahmer, E., & Avesani, C. (2002). Prosodic marking of Streeck, J. (1994). Gesture as Communication II: The Audience as Co Stec, K. (2012). Meaningful shifts. Meaningful K. (2012). Stec, Stokoe, W. C. (2001). Shattuck Shattuck Shattuck Seyfeddinipur, M., Kita, S., & Indefrey, P. (2008). How speakers interrupt ………… … …………

(pp. …

262. 262. – 232. – (2), 250 (2), 55 , , (57), 209 (57), ,

117 , 101225. 101225. , Speech Communication

64 Developmental Psychology Developmental ,

. Genova, Italy. Genova, . tures tures leads to better narrative discourse entals of speech synthesis and speech recognition speech and synthesis speech of entals 708. 708. – Proceedings Proceedings of the 9th International Conference on Speech , 704 , Fundam AN: AN: a Professional Framework for Multimodality Research. 62). Chichester:62). Wiley. John – Giménez, I., & Prieto, P. (2018). Encouraging children to produce Keller Keller (Ed.), 41 (2006). (2006). EL Proceedings of the 5th International Conference on Language Resources (LREC) Evaluation and Does Does restricting hand gestures Instruction and Learning impair mathematical reasoning? Prosody 2018Prosody overview. An interaction: rhythmic beat ges performances. discourse performance. discourse -

Zellner, B. (1994). Pauses and the temporal structure of speech. In E. Walkington, Walkington, C., Woods, D., Nathan, M. J., Chelule, G., & Wang, M. (2019). Wittenburg, P., Brugman, H., Russel, A., Klassmann, A., & Sloetjes, H. Wagner, Wagner, P., Malisz, Z., & Kopp, S. (2014). Gesture and speech in Vilà

118

………… … ………… by

…

Simon’s Simon’s Cat

. Appendix

ermission phase estraining gestures estraining

with p tudy 2

were adapted from S

119 below

Familiarization tudy 1 and S restraining gestures restraining gestures -

on N Study 1: Restraining Gestures The comic strips used for both studies Simon Tofield are reproduced and Materials for for Materials

gestures restraining -

on N

120 /

estraining gestures gestures estraining R

………… …

121 (Restraining (Restraining condition); Half of the participants

R the two groups of participants the groups two condition), condition),

Restraining - (Non

N kept the between same (N, R) explained half of the comic strips in the N condition, while the same comic strips were explained in the R condition by the other half of the participants. Condition order was Note:

phase ncouraging gestures ncouraging

122 Gestures Familiarization

encouraging gestures gestures encouraging

- on N Encouraging Study 2:

………… …

encouraging gestures encouraging -

123 /

ncouraging gestures gestures ncouraging

(Encouraging (Encouraging condition); Half of the

E 124 the two groups of participants. the groups two

Encouraging Encouraging condition), - (Non

………… … strips were explained in the E condition by the other half of the participants. Condition order kept was the same (N, E) between participants participants explained half of the comic strips in the N condition, while the same comic Note: ………… of

the . s

Paolo … Gibert emails) -

and and for and and the support,

up - for having

through the . . I would also work I’ve learned so for her help in

participated participated in the , Pilar, emails and piece (LCL). (LCL).

Anna Anna Bettoni

of that

research group from

and and guidance Annalisa Annalisa Oboe

hank you T reviewers, Núria Esteve director

people

and

the GrEP

Thank you, Grazia, for these years

, support . 125 supervisor, Pilar Prieto, whose Maria Maria Teresa Musacchio - ; ; the DiSLL’s my PhD technicians, with special mentions to experiments among her students. And I would a

of Rocco Rocco Coronato

her presence the

for their dedication to reading this

for Department Department of Linguistic and Literary Sciences (DiSLL) DiSLL’s

Language Language and Communication Lab directors

’s grateful to my co Graziano,

. o day. at the

gram also Maria Maria finding participants like to extend my gratitude to all other like to thank the Scanferlat I am particularly grateful to I am grateful to the of the University of Padov PhD pro I would also like to thank the dissertation and observations. their and precious comments writings. Last but not least, Barcelona thank that made you me feel for part of your warm firstvery welcome in were sometimes all that kept me going. believed in the project (and in me) from prompt the and detailed beginning feedback on and my thousands for your much along the way, and this wouldn’t have been this wouldn’t possible and you without the way, along much I am knowledge, and enduring optimism (together with her cheering First and foremost, I would like to express my gratitude to my supervisor Maria Grazia Busà for good and bad moments together Acknowledgements

. t De Page Diana

: Simone Antonio

help and I want to . ; Emanuela Emanuela Chiara

helped me . My My heartfel Virginia Zorzi

. ) in Padova

Thanks to . !) cooperating experience Roncen, Roncen, Valeria Russo,

that that I need to thank also

at at conferences that made My My thanks also go to

for the great assistance with ! the partnership PhD colleagues advice on voice acoustics voice on advice for

ding and acoustic analysis I have met what they were about. Thank you so

the best of luck and and 126

lovely Paolo Paolo Mairano the best first glance of what studying ; that

you to my with enthusiasm and motivation. So been very important to me and they or Boh Café

all about to to have shared this experience with

I wish ( is

for the statistics meetings Bertocchini, Bertocchini, Teresa Cancro, Camilla Covazzi,

my thanks Bar Bar Boh great colleagues the greatest endurance in listening to the same stories all Asli Asli Özyürek for

Nicolò Nicolò “The Listener” Marcato also by getting just some fun comics experience and a breakfast n pp group itself has been a wonderful language

s Pietro for meeting here and there and for sharing ideas, A

I never met her in person but her online Statistics & R course

nd nd extend extend for his inspiring seminars and and

I’m also happy

Giulia Giulia Nalesso, Olga Retziou, Francesco hats . A

W the interesting and relevant references he had always ready for me

keep going over the years list here some and and for spot advice on audio recor Romano, Human connections have Campisi multimodal for was very helpful along the way. helpful very was along I wish to express my gratitude now to various people that I enough to was meet over lucky the years and that inspired me so much and and Dario del Fante Gastaldon papers Piccinini. Our Our providing support and fun. I wish we went out for though! drinks more often, I wish to Antonello, Paoli, Giovanna Todaro, Raffaele Tondini, Angelica Vedelago, Susanna Zellini having show over again pretending not to know have done this you. without I wouldn’t much! (thanks (thanks to the thanks go to the piloting of the experiments and with the final data collection and for experiment ………… …

- ………… of to

s and and

Bjørn Bjørn

Laura … (Grup (Grup nice at at UPF

Federica so so badly,

Coromina, Coromina, - nez -

Joan Joan Borràs feel blessed to

Gimé Ingrid you have -

(together with all , , Kim Ouwehand,

& GrEP group

Teodor Sorodoc . Vilà - Also, I

and and its food was great to share with Judith Judith Llanes

Ionut Judith Judith at LCL at biscottini vegani dalle ragazze Florence Baills, coffee/cigarette breaks or for the nice moments we shared

design, design, statistics, R and all sort

Pilar,

And And to Andrea Liotto: it was . 127 Olga Olga Kushch, ent while, I realized I couldn’t have wished

want want to express my gratitude to colleagues colleagues in Padova, Tommaso Balsemin,

little s s we shared). Ps. for the great also Naoto Ienaga

LCL

I even from far away. It meant a lot to me and it to ).

the amazing people of the

in Barcelona:

Evi Kiagia,

I’m very grateful to all of you for always letting me

grazie tramezzini tramezzini al Nazionale

but, after that , , Manon Lelandais, Vito Evola, Marieke Hoetjes, Valentina

from the GLiF our our advice on experim h to thank

Anita Anita Slonimska and Luigi Palumbo: it with us at LCL. : send my

olvig Yuan Yuan Zhang. T - , Iris Hübscher, essel and our tapas nights too nights tapas our and Sara Sara Brugnerotto, Isabella Matticchio goes my deepest gratitude. It took a while (Sara?) Laura Aina ( To the dearest friends and the fun and lovely moment made your best to make me missing Catalunya but I’m grateful anyway. Also, I wish to thank Mariia (Marusia) Pronina, Patrick Rohrer, Ingrid Vincze, feeling part of the GrEP, has been a great source of motivation and enthusiasm Also, I wis d’Estudis de Prosòdia) Comes things things really made me “rolling more smoothly” towards the end of all this. Our chats eating the best are memories among years of the past I wish to during his research visit in Padova have you Cavicchio have met you the old times in Nijmegen (back in 2014) and the first approach to studies. gesture W De Iacovo, Mike Stevens, Matt Stave, Morgana Proietti Wim Pouw, James Trujillo, Lieke Van Maastricht. them so special and exciting (pretty much in order of appearance):

in tu ità. perché , come

. e Marco Costa

senza il vostro

sia per articoli , nonostante tutto. amore, sostegno e

durante durante il mio dottorato preclusa for for free amichetti

con con dolcezza e indubbia comic

to to sul pavimento di un’aula come

famiglia che è abbastanza grande da sul più bello re three in the years last nei momenti chiave. Grazie alla zia (2) la condivisione di dissidi sul piano rimanete

128 , , poggia

da da sempre

a a Barcelona Grazie a Rosanna, Renzo e Annica per esserci da quando quando può Alice Alice Spadaro, Salvo Sapienza/Arkee re

. amento). amento). A Puque che fa sempre il tifo per me , being always the being always

for

trova nche dei ringraziamenti per alcune cose più specifiche tra

che per la tesi stessa trovato trovato 8 anni fa dal dal principio e però (e a Luca che fa bene a fare sempre il tifo per lei). Allo zio

fatti

esservi better colleagues and friends than the three of you! You have been a

avresti avresti trovato monete, banconote e quadrifogli. Sei la mia fortuna più grande. Sono Sono grata a Sasy e Piciu per il loro essere nonni sempre a tempo pieno anche nella distanza otto anni come una famiglia da cui tornare. E poi volevo dire Sandro. Ti ho grazie a una mamma Marco che compare Cume per essere presente essere la mia personale comprensione (chi fonte sa inesauribile quante cose di supporto e incoraggi mi sarei vita. Sono immensamente grata alla mia farmi sentire sommersa di affetto. A Mamma e Papà (Egle e Rino) per sapete tutto Cioffee merita a le quali (1) le consulenze sull’Inglese ottenute fase di stesura Un grazie speciale ad per (e per mille altre cose che già sapete). Fab, Cioffee, Tosa e Chiara And And I’ll switch to Italian now, to thank all the people who have known me for the better part of my life and have made it somewhat easier to get to this today. point mix of friendship and support service that was mostly achieved via drinks, food and endless laughs (or audio messages). In short, thank you Tb, Sara, Paola!) (and Isa for ………… …