Marketing Science Institute Working Paper Series 2020 Report No. 20-104 A Natural Language Processing Approach to Predicting the Persuasiveness of Marketing Communications Siham El Kihal, A. Selin Atalay, and Florian Ellsaesser “A Natural Language Processing Approach to Predicting the Persuasiveness of Marketing Communications” © 2020 Siham El Kihal, A. Selin Atalay, and Florian Ellsaesser MSI working papers are distributed for the benefit of MSI corporate and academic members and the general public. Reports are not to be reproduced or published in any form or by any means, electronic or mechanical, without written permission. 1. Introduction The use of language is ubiquitous in any marketing message communicated such as blogs, emails or product descriptions, and so is the decision problem how to formulate such messages so that they are effective with a wide audience. Most marketing messages are crafted with the goal of facilitating persuasion. Persuasion, changing individuals’ thoughts, feelings, or behavior (Rocklage et al. 2018) is a major outcome sought in marketing communications. Language, spoken and written alike, is the fundamental tool used to express a message intended for persuasion. With the growth of the internet and connectivity, companies today have many tools at their disposal that they can instrumentalize to spread various lengths of messages for numerous purposes. For instance, the use of email, social media posts, live chats, and blogs provide unprecedented opportunities to reach consumers with longer messages than those that can be used to reach consumers on traditional channels such as TV, radio or newspaper ads. Many companies are trying to make use of these opportunities. Recently, Brandwatch estimated that there are more than 60 million active business pages on Facebook attempting to reach their customers (Brandwatch 2019). The verbal content created online, per minute is immense: for example, using the internet, per minute 12,989,111 text messages are sent; 473,400 messages are tweeted on Twitter; 79,740 blogs are posted on Tmblr; pointing to a rising need to better evaluate which message will perform better and be more persuasive (Statista 2018). The aim of the current research is to understand how the use of language impacts persuasion. We answer the following research question: What is the role of language in predicting how persuasive a message will be? Our goal in asking this question is twofold. First, we want to contribute to the literature in persuasion by testing the unique role of language. Second, we want to provide guidance to decision makers or those that design marketing messages by developing a tool they can use to formulate their messages and/or assess the persuasiveness of their messages. To answer this question, we focus on the two main elements of language that comprise each message: (1) The choice of words used to communicate the content of the message which is referred to as diction. (2) The arrangement of the words creating the sentences, which is referred to as syntax. 1 Marketing Science Institute Working Paper Series Syntax is the grammatical structure of the words communicating the intended meaning of the sentences. Psycholinguistic studies have established that both diction and syntax are crucial for language processing (Bates 1995). Thus, we expect both these elements to impact how individuals process a message, and thereby impact how persuasive that message is. Building on this understanding, we develop a machine learning approach to predict the persuasiveness of messages as a function of diction and syntax. We use a dataset including 134 debates with 129,480 sentences and a follow-up experiment to measure content and syntactic complexity and predict the persuasiveness of messages. More specifically, we use the LIWC dictionary (Pennebaker et al. 2015) to classify the categories of words used in the message and measure content complexity. We take a natural language processing approach and use universal dependency parsing based on convolutional neural networks to classify the syntax of the sentences in the message and measure syntactic complexity. The measurement of syntactic complexity in larger corpora of language has only recently become possible with advances in natural language processing (NLP). An emerging stream of research in marketing has used NLP to improve decision-making. For instance, to analyse user-generated content to identify consumer content preferences, predict future changes in sales, or analyse market structures (Archak et al. 2011, Lee and Bradlow 2011, Liu and Toubia 2018, Netzer et al. 2012). Recently, Timoshenko and Hauser (2019) used advanced NLP methods to identify customer needs from user-generated content as well as to identify new opportunities in marketing contexts such as new product development. Our approach combines advances in NLP with traditional dictionary approaches to investigate how both diction and syntax contribute to predicting the persuasiveness of a message. We show that syntax can be used to predict how persuasive any message will be, beyond diction. 2. Theoretical Background: Language and Persuasion The Elaboration Likelihood Model (ELM, Petty and Cacioppo 1986, Petty and Wegener 1999) provides a comprehensive account of elements that impact attitude change and persuasion. The ELM is a dual-route theory, which explains that the message, source (i.e., speaker) and context elements of a communication are processed differently in different contexts to facilitate changes in judgment. 2 Marketing Science Institute Working Paper Series Language is part of the message, and the ELM points to how language may be processed by different individuals in different contexts. The ELM suggests that the motivation and ability of the audience are two factors that are instrumental to predicting how much the audience will be willing or able to elaborate on a message to accept or reject it. The motivation and the ability to elaborate on a message may be due to factors related to the audience (i.e. cognitive skills, involvement), the context (i.e., distractors are present, message is repeated) or to the message itself (i.e., number of arguments, complexity of arguments, comprehensibility of content). According to the ELM, the impact of a persuasive message is differentiated by the degree of elaborative information processing activity, which is determined by the motivation, and ability of the audience. The ELM suggests that individuals who are motivated and able to elaborate on the message presented respond differently to complex messages, compared to those individuals that are not motivated or able to elaborate on the message presented. Complex messages may be harder to comprehend compared to simple messages due to limited processing abilities or when the motivation to process is low. From the perspective of ELM, this may mean that complex messages will be more persuasive than simple messages for those individuals that are motivated and able to deliberately process the message, because these individuals will be able to process the content and evaluate the claims made to form their judgments. Complex messages can also be persuasive for those individuals that are not motivated or able to deliberately process the message. Through heuristic processing, these individuals may make inferences (i.e., credibility, expertise) about the information and be nonetheless be persuaded by it. Note that this view equates message complexity with content complexity. From the perspective of linguistics, there is however, another dimension of message complexity. This dimension is the syntactic complexity, which is the complexity of the grammatical structure of the words in the message. A message that is not complex in terms of content can still be complex in terms of its syntax. Syntactic complexity impacts readability, reaction times, and recall (c.f., Lowrey 1998, Lowrey 2006, Bradley and Meeds 2002), making it at least as critical as content complexity for the persuasiveness of 3 Marketing Science Institute Working Paper Series a message. We expect syntactic complexity to affect the persuasiveness of a message from an information processing perspective. In order to process any syntax, working memory is required such that the linguistic material can be temporarily simultaneously stored and processed to extract its meaning (Gibson 1998, Lewis et al. 2006). Working memory is the limited resource used for information processing. It is the mental capacity individuals have to allocate to information processing tasks such as reasoning, comprehension, and learning (Cowan 2010, Engle 2001, Britton et al.1982, Bradley and Meeds 2002). If there is not enough mental capacity, proper information processing is not feasible (Basil 1994, Schneider et al. 1984, Shiffrin and Schneider 1977, Lang 2000, Lang 2006). The limited capacity of working memory that impacts learning tasks such as serial recall, also impacts language processing (McElree 2006, Cowan 2001). To comprehend sentences and the overall meaning of a message, individuals have to process syntax, through a mechanism referred to as sentence parsing. The process of sentence parsing refers to how individuals break down a sentence into its grammatical components, and identify the syntactic relationships between words. Sentence parsing is an automatic process that individuals engage in without thinking about it. The syntactic relations, the links between the words, are referred to as dependencies (Nivre et al. 2016, McDonald