Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models

Decoding Brain Activity Associated with Literal and Metaphoric Sentence Comprehension Using Distributional Semantic Models Vesna G. Djokic† Jean Maillard‡ Luana Bulat‡ Ekaterina Shutova† †ILLC, University of Amsterdam, The Netherlands ‡Dept. of Computer Science & Technology, University of Cambridge, United Kingdom [email protected], [email protected], [email protected], [email protected] Abstract ability of distributional models to predict patterns Recent years have seen a growing interest of brain activity associated with the meaning of within the natural language processing (NLP) words, obtained via functionalmagnetic resonance community in evaluating the ability of seman- imaging (fMRI) (Mitchell et al., 2008; Devereux tic models to capture human meaning represen- etal., 2010; Pereiraet al.,2013).Following in their tation in the brain. Existing research has mainly steps, Anderson et al. (2017b) have investigated focused on applying semantic models to de- visually groundedsemantic models in this context. code brain activity patterns associated with They found that while both visual and text-based the meaning of individual words, and, more models can equally decode concrete words, text- recently, this approach has been extended to based models show an overall advantage over vi- sentences and larger text fragments. Our work sual models when decoding more abstract words. is the first to investigate metaphor process- Other research has shown that data-driven ing in the brain in this context. We evaluate semantic models can also successfully predict pat- a range of semantic models (word embed- terns of brain activity associated with the pro- dings, compositional, and visual models) in cessing of sentences (Pereira et al., 2018) and their ability to decode brain activity associated larger narrative text passages (Wehbe et al., 2014; with reading of both literal and metaphoric Huth et al., 2016). Recently, Jain and Huth (2018) sentences. Our results suggest that composi- investigated long short-term memory (LSTM) tional models and word embeddings are able recurrent neural networks and showed that seman- to capture differences in the processing of lit- tic models that incorporate larger-sized context eral and metaphoric sentences, providing sup- windows outperform those with smaller-sized port for the idea that the literal meaning is context windows, as well as the baseline bag- not fully accessible during familiar metaphor of-words model, in predicting brain activity comprehension. associated with narrative listening. This suggests that compositional semantic models are suffi- ciently advanced to study the impact of linguistic 1 Introduction context on semantic representation in the brain. In Distributionalsemanticsaims to represent the mean- this paper, we investigate the extent to which lex- ing of linguistic fragments as high-dimensional ical and compositional semantic models are able dense vectors. It has been successfully used to to capture differencesin human meaning represen- model the meaning of individual words in seman- tations, resulting from meaning disambiguation of tic similarity and analogy tasks (Mikolov et al., literal and metaphoric uses of words in context. 2013; Pennington et al., 2014); as well as the Metaphoric uses of words involve a transfer meaning of larger linguistic units in a variety of of meaning, arising through semantic composition tasks, such as translation (Bahdanau et al., 2014) (Mohammad et al., 2016). For instance, the mean- and natural language inference (Bowman et al., ing of the verb push is not intrinsically metaphor- 2015). Recent research has also demonstrated the ical; yet it receives a metaphorical interpretation 231 Transactions of the Association for Computational Linguistics, vol. 8, pp. 231–246, 2020. https://doi.org/10.1162/tacl a 00307 Action Editor: Walter Daelemans. Submission batch: 11/2019; Published 4/2020. c 2020 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license. Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00307 by guest on 30 September 2021 when we talk about pushing agendas, push- word-based models, namely, word embeddings of ing drugs, or pushing ourselves. Theories of the verb and the nominal object; (2) compositional metaphor comprehension differ in terms of the models, namely, vector addition and an LSTM kinds of processes (and stages) involved in ar- recurrent neural network; and (3) visual models, riving at the metaphorical interpretation, mainly learning visual representations of the verb and whether or not the abstract meaning is indirectly its nominal object. This choice of models allows accessed via processing the literal meaning first us to investigate: (1) the role of the verb and or directly accessible largely bypassing the lit- its nominal object (captured by their respective eral meaning (Bambini et al., 2016). To this ex- word embeddings) in the interpretation of literal tent, the role that access to and retrieval of the and metaphoric sentences; (2) the extent to which literal meaning plays during metaphor processing compositional models capture the patterns of hu- is often debated. On the one hand, metaphor com- man meaning representation in case of literal and prehension involves juxtaposing two unlike things metaphoric use; and (3) the role of visual infor- and this may invite a searchfor common relational mation in literal and metaphor interpretation. We structure through a process of direct comparison. test these models in their ability to decode brain Inferences flow from the vehicle to the topic giv- activity associated with literal and metaphoric ing rise to the metaphoric interpretation (Gentner sentence comprehension, using the similarity de- and Bowdle, 2005). In a slightly different vein, coding method of Anderson et al. (2016). We per- Lakoff (1980) suggest that metaphor comprehen- form decoding at the whole brain level, as well sion involves systematic mappings (between a as within specific regions implicated in linguistic, concrete domain onto another typically more ab- motor and visual processing. stract domain) that become established through Our results demonstrate that several of our se- co-occurrences over the course of experience. mantic models are able to predict patterns of brain This draws on mental imagery or the re-activation activity associated with the meaning of literal and of neural representations involved during pri- metaphorical sentences. We find that (1) com- mary experience (i.e., sensorimotor simulation) positional semantic models are superior in decod- allowing appropriate inferences to be made. Other ing both literal and metaphorical sentences as theories, however, suggest that the literal mean- compared to the lexical (i.e., word-based) mod- ing in metaphor comprehension may be largely els; (2) semantic representations of the verb are bypassed if the abstract meaning is directly or superior compared to that of the nominal object in immediately accessible involving more categori- decoding literal phrases, whereas semantic repre- cal processing (Glucksberg, 2003). For example, sentations of the object are superior to that of the the word used metaphorically could be imme- verb in decoding metaphoricalphrases;and (3)lin- diately recognized as belonging to an abstract guistic models capture both language-related and superordinate category of which both the vehi- sensorimotor representations for literal sentences— cle and topic belong. Alternatively, it has been in contrast, for metaphoric sentences, linguistic suggested that more familiar metaphors involve models capture language-related representations categorical processing, while comparatively novel and the visual models captured sensorimotor rep- metaphor will involve initially greater processing resentations in the brain. Although the results do of the literal meaning (Desai et al., 2011). not offer straightforward conclusions regarding To contribute to our understanding of metaphor the role of the literal meaninginmetaphor compre- comprehension, including the accessibility of the hension,they provide some supportto the idea that literal meaning, we investigate whether semantic lexical-semantic relations associated with the lit- models are able to decode patterns of brain activ- eral meaning are not fully accessible during famil- ity associated with literal and metaphoric sentence iar metaphor comprehension, particularly within comprehension, using the fMRI dataset of Djokic action-related brain regions. et al. (forthcoming). This dataset contains neural activity associated with the processing of both 2 Related Work literal and familiar metaphorical uses of hand- 2.1 Decoding Brain Activity action verbs (such as push, grasp, squeeze, etc.) in the context of their nominal object. We exper- Mitchell et al. (2008) were the first to show that iment with several kinds of semantic models: (1) distributional representations of concrete nouns 232 Downloaded from http://www.mitpressjournals.org/doi/pdf/10.1162/tacl_a_00307 by guest on 30 September 2021 built from co-occurrence counts with 25 experi- features obtained from an LSTM language model ential verbs could predict brain activity elicited by outperformed the bag-of-words model. Moreover, these nouns. Later studies used the fMRI data of the performance increased when using LSTMs Mitchell et al. (2008) as a benchmark for testing a with larger context-windows. range of semantic models including topic model- Inparalleltothis work,severalother workshave based semantic

Load more