Modeling affirmative and negated action processing in the brain with lexical and compositional semantic models Vesna G. Djokic♠ Jean Maillard| Luana Bulat| Ekaterina Shutova} ♠Department of Neuroscience, University of Southern California, USA |Dept. of Computer Science & Technology, University of Cambridge, United Kingdom }ILLC, University of Amsterdam, The Netherlands [email protected], [email protected], [email protected], [email protected] Abstract ability of DSMs to predict fMRI patterns of sen- tential meanings (Pereira et al., 2018) and larger Recent work shows that distributional seman- narrative text passages (Wehbe et al., 2014; Huth tic models can be used to decode patterns of brain activity associated with individual words et al., 2016). They have shown that encoding mod- and sentence meanings. However, it is yet un- els based on word embeddings are able to capture clear to what extent such models can be used subtle aspects of sentence meaning in the brain, to study and decode fMRI patterns associated even when these models are oblivious of word or- with specific aspects of semantic composition der and syntactic structure. While promising, none such as the negation function. In this paper, of this research has so far systematically investi- we apply lexical and compositional seman- gated specific semantic composition phenomena tic models to decode fMRI patterns associated and processing at the syntax-semantic interface, with negated and affirmative sentences con- taining hand-action verbs. Our results show such as that of the negation function. reduced decoding (correlation) of sentences Negation is a fundamental abstraction necessary where the verb is in the negated context, as for efficient reasoning and communication (Horn, compared to the affirmative one, within brain 1989). Although it is typically marked syntac- regions implicated in action-semantic process- tically, the semantics of negation in natural lan- ing. This supports behavioral and brain imag- guage usage has proven to be rather challenging ing studies, suggesting that negation involves reduced access to aspects of the affirmative to pinpoint (Speranza and Horn, 2010). In logi- mental representation. The results pave the cal negation, the negation operator has been suc- way for testing alternate semantic models of cinctly described as a truth-functional operation, negation against human semantic processing reversing the truth value of a sentence. On the in the brain. other hand, from a pragmatic point of view, the primary function of negation is to direct attention 1 Introduction to an alternative meaning and can thus be, more Computational semantic models are increasingly generally, compared to our ability for counterfac- being evaluated in their ability to capture aspects tual thinking (Hasson and Glucksberg, 2006). It is of human semantic processing, including similar- also often assumed that negation entails affirma- ity and association judgments (De Deyne et al., tion (as it is always positive by default), yet the 2016) and semantic representation in the brain extent to which the the affirmative situation need (Bulat et al., 2017). Prior work shows that dis- be processes is debated (Orenes et al., 2014). De- tributional semantic models (DSMs) are able to spite the intuition that negated meanings are in- decode functional magnetic resonance imaging deed quite distinct from their affirmative counter- (fMRI) patterns associated with the meaning of parts, there is still no comprehensive account of concrete words (Anderson et al., 2013). Relevant how the brain represents negated entities. to our work, Carota et al.(2017) showed that the Neuroscientific studies on negation have pre- similarity structure of DSMs for action words cor- dominantly focused on studying negation of relates with that of fMRI patterns in brain regions action-related sentences and suggest that nega- implicated in action-semantic processing. tion blocks access to aspects of the affirmative More recent studies have also investigated the representation (Papeo et al., 2016). For exam- 5155 Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5155–5165 Florence, Italy, July 28 - August 2, 2019. c 2019 Association for Computational Linguistics ple, negation of action-related sentences or im- a reduced correlation between the similarity struc- peratives involves decreased activity in motor sys- ture of DSMs of action verbs and fMRI patterns tems of the brain implicated in action semantics of negated as compared to affirmative action sen- when compared to the affirmative context (Tetta- tences. Importantly, we show for the first time that manti et al., 2008; Tomasino et al., 2010). How- negation impacts semantic similarity in motor ar- ever, overall reduced activation does not necessar- eas, but also to some extent language-related brain ily equate to a lack of information across patterns regions. These findings lend further support to the of activated or deactivated voxels in a brain region hypothesis that negation may involve reduced ac- (Kriegeskorte et al., 2008). More importantly, the cess to aspects of the affirmative mental represen- degree to which negation of action-related sen- tation. tences impacts access to lexico-semantic repre- sentations and semantic similarity in the brain is 2 Related Work not yet well understood. To contribute to our un- derstanding of negation and its modeling, we in- Decoding brain activity Mitchell et al.(2008) vestigate the extent to which lexical and compo- were the first to show that DSMs based on co- sitional semantic models can decode fMRI pat- occurrence counts with 25 sensorimotor verbs terns of negated and affirmative action sentences (e.g. see, hear, taste) can predict fMRI pat- in the brain using similarity-based decoding (An- terns associated with the meaning of concrete derson et al., 2016). We also test the extent nouns. Later research has demonstrated that a to which the representational similarity structure range of DSMs can decode fMRI patterns of con- (Kriegeskorte et al., 2008) of DSMs of action- crete nouns (Murphy et al., 2012; Anderson et al., verbs correlates with that of fMRI patterns asso- 2013; Bulat et al., 2017) and, more recently, ab- ciated with negated versus affirmative sentences stract nouns (Anderson et al., 2017). Most rel- containing hand-action verbs. We focus on motor evant to our study, Carota et al.(2017) showed areas and classical language-related brain regions that the similarity structure of a Latent Seman- implicated in action-semantic processing (e.g., un- tic Analysis (LSA) model for action words (nouns derstanding action words and sentences) (Pulver- and verbs) correlates with that of fMRI patterns in muller, 2005; Kemmerer, 2015). motor areas (left precentral gyrus (LPG)) and clas- sical language-related brain regions (left inferior DSMs have proven successful in modeling as- frontal gyrus (LIFG), left posterior middle tempo- pects of semantic composition in the context ral gyurs (LMTP)) implicated in lexico-semantic of the natural language inference task (Bowman processing (Binder et al., 2009). et al., 2015b). Although the modeling of logical Moving beyond words, other studies have negation using DSMs is wrought with challenges shown that DSMs can predict brain activity pat- (Kruszewski et al., 2017), current state-of-the-art terns associated with larger linguistic units (We- neural network based models appear to capture el- hbe et al., 2014; Huth et al., 2016; Pereira et al., ements of markedness asymmetry in negation (Li 2018). For example, Pereira et al.(2018) showed et al., 2016) and, presumably, implicitly model that a regression model mapping between fMRI negation at some level. In our experiments, we patterns of words and their word embeddings investigate the extent to which DSMs are able could synthesize vector representations for novel to decode (correlate with) fMRI patterns asso- sentences that correlate with the average of the ciated with the reading of sentences containing word embeddings of the sentence. Working with negated and affirmative action verbs. We exper- larger text fragments, Wehbe et al.(2014) and iment with (1) word-level representations of ac- Huth et al.(2016) have been able to predict neu- tion verbs; and (2) compositional semantic models ral activity associated with the processing of narra- (based on addition of word-level representations tives in the brain using encoding models with word and long short-term memory (LSTM) networks). embeddings (also syntactic markers) as features. In agreement with previous work, our results Although these findings suggest that DSMs are show that distributional representations of action able to predict fMRI patterns associated with the verbs (and to some extent verb-object phrases) processing of compositional meanings, they do not show reduced decoding for negated versus affir- reveal to what extent the models capture specific mative action sentences. This is also reflected as compositional phenomena nor the specific impact 5156 of linguistic context on semantic representation in to prior studies suggesting a link between action the brain. Our work extends this line of research to negation and the inhibition of actions (de Vega study individual aspects of semantic composition, et al., 2016). focusing on the negation function. These findings seem in some regards contrary to Modeling negation in NLP Kruszewski et al. the predictions of linguistic theories of negation. (2017) contrast logical negation, which captures