Filozofická fakulta Univerzity Palackého
SUBSTITUTIVE EMOJIS
(Diplomová práce)
2019 Michelle Martinek
Filozofická fakulta Univerzity Palackého Katedra anglistiky a amerikanistiky SUBSTITUTIVE EMOJIS (Diplomová práce)
Autor: Michelle Martinek Studijní obor: Anglická filologie Vedoucí práce: Mgr. Ondřej Molnár, Ph.D. Počet stran: 80 Počet znaků: 122,743 Olomouc 2019
Prohlášení
Prohlašuji, že jsem diplomovou práci vypracovala samostatně a použila jen uvedenou literaturu a zdroje.
V Olomouci dne…………………… Podpis…………………………
Poděkování Děkuji Mgr. Ondřeji Molnárovi, Ph.D. za odborné vedení při tvorbě diplomové práce. Abstract
This aim of this thesis is to investigate the possibility of emojis substituting language. Marcel Danesi outlines two functions of emojis in his book The Semiotics of Emoji: adjunctive and substitutive. Adjunctive emojis, which only complement the text, are the topic of a few publications and studies. However, when it comes to substitutive emojis, which replace words in text, there is incredibly little literature and no studies researching them in practice. The thesis begins with briefly reviewing the history of emojis and existing literature on emojis to identify a gap in knowledge. After this, other similar visual forms of script are discussed to determine the potential benefits and limitations of substitutive emojis. Following this is the practical part of the thesis, which is an analysis of over 900 examples of substitutive emojis from Instagram to better understand how they are used with text and why.
Key words Emoji, substitution, emoticons, computer-mediated communication, multimodality
Anotace
Záměr diplomové práce je zkoumat emoji jako náhradu textu. V knize The Semiotics of Emoji Marcel Danesi upřesňuje dva způsoby návaznosti emoji na text. První způsob doplňuje, co je už vyjádřeno textem (anglicky adjunctive emojis). Na toto téma je k dispozici několik studií. Naopak studie nejsou žádné u druhého způsobu, kdy emoji text nahrazuje (anglicky substitutive emojis). Diplomová práce začíná přehledem historie emoji a existující literatury o nich. Jejím cílem je najít, analyzovat a doplnit chybějící informace o emoji, které nahrazují text. Poté jsou probírány další podobné grafické znaky vizuální komunikace, aby se zjistily možné výhody a omezení těchto emoji. Praktická část analyzuje více než 900 příkladů zmíněných emoji z Instagramu a umožňuje nám lépe rozumět, jak jsou používané a proč.
Klíčová slova Emoji, náhrada, emotikony, počítačem zprostředkovaná komunikace, multimodalita
Contents 1 Introduction ...... 1 2 Defining Emojis ...... 3 2.1 Development of the Emoji ...... 4 2.2 Why Use Emojis? ...... 6 3 Review of Existing Literature ...... 8 3.1 Defining Adjunctive and Substitutive ...... 8 3.2 Emoticons ...... 9 3.3 Emojis ...... 11 3.4 Identifying the Gaps in Knowledge ...... 14 4 Images Substituting Text ...... 15 4.1 Pictographs ...... 15 4.2 Rebus ...... 17 4.3 Emoji-Only Communication ...... 18 4.4 Mixed Textuality: A Multimodal Approach ...... 23 4.5 The Plausibility of Substitutive Emojis ...... 24 4.6 Hypotheses to Test ...... 25 5 Substitutive Emojis on Instagram ...... 27 5.1 Methodology ...... 27 5.2 Results ...... 29 6 Analysis ...... 34 6.1 Parts of Speech ...... 34 6.1.1 Nouns ...... 35 6.1.2 Verbs ...... 39 6.1.3 Adjectives ...... 42 6.1.4 Adverbs ...... 46 6.1.5 Other Parts of Speech ...... 46 6.1.6 In Conclusion of Parts of Speech ...... 48 6.2 Emoji Categories ...... 49 6.2.1 Face ...... 49 6.2.2 Food & Drink ...... 51 6.2.3 Objects ...... 51 6.2.4 People & Body ...... 52 6.2.5 Emotion ...... 53 6.2.6 Symbols ...... 54 6.2.7 In Conclusion of Emoji Categories ...... 54 6.3 Miscellaneous Observations ...... 55 6.3.1 Motivation for Emoji Substitution ...... 56 6.3.2 Message Composition ...... 58 6.3.3 Selected Examples of Unique Emoji Substitutions ...... 60 7 Conclusion ...... 62 8 Data Tables ...... 64 9 Examples ...... 67 10 Resumé ...... 79 11 References ...... 76 1 Introduction
Over the past 30 years technology has significantly changed how we communicate. Computer-mediated communication has connected people from across the globe and made long-distance interaction nearly instantaneous. At first the communication was simple and only one mode of communication was involved, like text in SMS or instant messaging. Over time users were given the ability to add other forms of expression to their messages. For example, users could use text and images together to convey ideas. Adding these other forms of expression made the communication multimodal. There is one form of modality that was not standardized in online communication until more recently: emojis. Emojis are small colorful images that can be added to text messages, like a smiley face ( ). In 2011 emojis exploded in popularity thanks to updated software on mobile phones which made them more accessible to users. Despite the popularity of emojis, they are a relatively new phenomenon and not much is known about them from a linguistic standpoint. There is one prominent publication from Marcel Danesi that tried to fill in this knowledge, called The Semiotics of Emoji. In this book he mentions that there are two ways of using emojis with text. The first is adjunctively to text and the other as a substitute of text. Adjunctive emojis are used to complement information that is already given through text. They do not contribute any denotative meaning to a message. There are numerous studies on adjunctive emojis, especially focused on their ability to act as illocutionary force indicators, guiding the reader’s interpretation of a message. Researchers found that they act like emotional punctuation and resemble paralinguistic features of language. Substitutive emojis, on the other hand, replace text. While there are several studies focused on the adjunctive use of emojis, there are no publications nor available data focused on substitutive emojis. It is generally acknowledged that emojis can be used to substitute words, but there is no information discussing how they replace language. This is the gap in knowledge that this thesis primarily concerns itself with. The goal of this thesis is to collect data to fill in knowledge of substitutive emojis and determine how these emojis are used in practice. After reviewing phenomena similar to substitutive emojis, some possible benefits and limitations of using substitutive emojis are determined. Then a prediction is made about which parts of speech are most suitable for substitution. In addition to researching what parts of language are replaced, which emojis are used for substitution will also be examined. Previous research suggests that the face emojis, which feature facial expressions, are primarily used as a form of paralanguage or resemble nonverbal communication. However, face emojis make up only a fraction of existing emojis. Hundreds of other emojis feature things like objects and activities. Research from Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič suggests that these face emojis are used differently from all the other emojis. More specifically, the results hint that the face emojis tend to be used adjunctively while the other emojis are primarily used substitutively. Collecting data on which emojis are used in substitution will help to determine if this is true. The first few chapters of this thesis discuss the theoretical background of emojis. It begins with this introduction, which briefly outlines the topics of the thesis to the reader. Following this introduction is chapter 2, which reviews the history of emojis and a short discussion on why they were created. Chapter 3 then reviews existing literature on emojis to identify the gap in knowledge. Before collecting data to fill in that knowledge, chapter 4 discusses other similar visual forms of language to determine the potential benefits and limitations of substitutive emojis. After the chapters concerning background theory is the practical part of the thesis. The second half of the thesis is focused on collecting data on substitutive emojis and analyzing examples of them in practice. First a corpus is compiled of over 900 examples of substitutive emojis from Instagram. An analysis of the examples is conducted to better understand how the emojis are used with text and why. Most importantly, this data and analysis also confirms or denies the hypotheses regarding what language is substituted most and which emojis are used.
2
2 Defining Emojis
First, we must familiarize ourselves with what emojis are. A short review of their origins and development will show why emojis were created and how they serve users in computer mediated communication. The term emoji is a loanword from Japanese. The beginning e (絵) means “picture” and moji (文字) approximately means “written character” (Unicode Consortium 2018). They are small color images that can be used with text in computer mediated communication. Some of the most frequently used emojis show a human facial expression like a smile, wink, or frown, as can be seen below in Figure 1.
Figure 1: The ten most popular emojis used on Apple devices (Apple 2017)
The organization responsible for adding new emojis and maintaining the emoji inventory is the Unicode Consortium. The Consortium creates a unique character for each emoji, and that character is then converted into an image, like the ones shown above. The inventory of emojis also is not fixed but regularly expanded. The latest emojis were released with version 12.1 of the Unicode character set, which launched on October 21st, 2019, containing 1,262 unique emojis (Unicode Consortium 2019). Although emojis with facial expressions may be the most recognizable, emojis depict a wide variety of objects and concepts. The Unicode Consortium has made several categories for organizing their emojis, and even further specific subcategories, which illustrate their wide variety (refer to Table 1 on page 4). An even bigger variety can be achieved through modifiers. Using these modifiers a user can change the gender, skin tone, and hair color of a person depicted in an emoji. For example, the couple emoji defaults to showing a male and female standing together, but with modifiers the genders can be changed, making the emoji represent a same-sex couple. Emojis were popularized worldwide in 2011 after being integrated into Apple’s iOS software, making them a relatively new phenomenon (Pardes 2018; Schenker 2016). However, their origins lie in the 1990s, long before their popularization. Before discussing the origin of emojis, it is worthwhile to 3 talk about emoticons, which are their predecessor first used in the 1980s. (Pardes 2018) (Schenker 2016) – This is only here so word would pick up the above the citations Unique Emoji categories Examples of Subcategories Emojis
Smileys & Emotion Face-smiling, face-costume, emotion… 148 People & Body Hands, body-parts, person-gesture… 137 Animals & Nature Mammal, bug, plant-other… 126 Food & Drink Fruit, prepared, Asian, dishware… 121 Travel & Places Place-religious, transport-ground, sky & 210 weather… Activities Event, sport, game 79 Objects Clothing, office, household, light & 233 video… Symbols Warning, arrow, religion, zodiac, 203 geometric… Flags Country, subdivisions, other… 5
Table 1: The categories and some subcategories of emojis given by the Unicode Consortium. A full list can be found at unicode.org/emoji/charts/full-emoji-list.html. Unique emoji counts do not include emojis with modifiers.
2.1 Development of the Emoji
Before emojis exploded in popularity in 2011, emoticons were a common occurrence in computer-mediated communication. Emoticons are specific configurations of letters and punctuation which represent facial expressions, examples being :D and :) (Crystal 2006, 36). There are only a small few that are not facial expressions, like the heart symbol emoticon <3. The first recorded instance of an emoticon was used by Scott Fahlman in 1982, who proposed using the :-) and :-( emoticons to prevent misinterpretations of text. Later Fahlman explained that the smile was a mark showing a statement should be interpreted as a joke and the frown for statements that were meant to be taken seriously (Garrison, et al. 2011). Fahlman’s goal was to make meta- communicative symbols that could then be used universally (Gruber 2013).
4
Throughout the 1980s and 1990s emoticons grew in popularity as a feature in computer-mediated communication. At the end of the twentieth century we can see development of their successor: emojis. Japanese telecommunication planner Shigetaka Kurita first made the series of iconic images in 1999 (Moschini 2016). Emoticons largely only depicted simplified facial expressions and the primary purpose of the new emojis was to convey more complex concepts beyond expressions. Kurita’s efforts resulted in a mobile Internet platform called “i-mode” (Blagdon 2013). The first set of emojis was made up of 176 symbols, each using a 12-by-12 grid of pixels for the image, as illustrated in Figure 2. Like its emoticon predecessor, the emoji set included facial expressions but also objects such as a telephone, car, and high-heeled shoes. Now instead of using words for the weather, like “sunny”, an easy-to-read sun emoji could be used.
Figure 2: The first set of emojis, created by Shigetaka Kurita (Cherelus 2016)
After the success of Kurita’s emojis in Japan, Google became interested in the icons and tried to standardize them in 2006 (Busche 2016). The Unicode Consortium then took the next steps in standardizing emojis and expanded the original set of 176 symbols to just over 700 in 2009 (Evans 2017), which has grown overtime to the 1,262 emojis at present. Emojis have come a long way from their beginnings, and today it is estimated that over 90 percent of the world’s internet users make use of emojis on social media applications (Emogi Research Team 2015).
5
2.2 Why Use Emojis?
Once emojis were implemented and made regularly available, users were no longer limited to just communicating through text. These small colorful images let users add a new dimension to their messages. Over time social media platforms made them easier to input, allowing users to choose emojis through menus, and they became a quick and convenient way of making a message multimodal. A multimodal text means that the communication involves “two, three, or four dimensions”, using more than just language (speech or writing) to communicate and make meaning (Kress 2012, 36). This means that when reading a message, the emojis should be considered a part of the whole communication, not only the text, in order to fully understand the writer’s intention. There are a few reasons why a user would want to make their message multimodal. The adage “a picture is worth a thousand words” is a concise way of putting it; there are some situations where an emoji can sum up a large amount of text, getting the point across more succinctly. In other cases it may be that emojis deliver meaning and stimulate the reader in a way that text simply cannot. There is no doubt that colorful images come across differently to readers than black text. When interviewing users about emojis, Marcel Danesi found that they were preferred because they are “an efficient shorthand method, a fun and entertaining way of communicating a larger meaning in simpler ways.” (Danesi 2017, 115-116). The informants also often cited that they made communication more fun and relaxed compared to text alone. However emojis are not the only way to add modality to online messages. Long before emojis, many social platforms allowed users to include sound and video through programs like Skype. Many forms of online interaction also already supported the ability to share static and moving images like GIFs, which can add a similar effect like the emojis. There are a few reasons why a user would choose to use an emoji over other forms of modality. One of the biggest benefits is in input and selection. Most social media platforms have an integrated easy-to-use menu for searching, selecting, and inputting emojis into messages. Emojis are also
6 standardized thanks to the Unicode Consortium, meaning that they largely behave the same across social media platforms and devices, allowing for convenient and accessible selection menus. Images and animated GIFs on the other hand take more time since they are not standardized, requiring the user to go through more effort to find and execute them in their messages. However, in some situations a user may find that the extra effort is worth it because the GIF image captures the meaning they want to communicate more accurately. Other modes like sound and video can be problematic for the user to incorporate into their message because of the limits of social media platforms, making the emojis much more convenient. Emojis add another convenient form of modality to messages, making them multimodal, and contribute a unique meaning to messages that may not have been achieved as concisely through text alone, if at all. What emojis exactly contribute to language is a topic that several publications have considered and will be the focus of the next section of this thesis.
7
3 Review of Existing Literature
Emojis have exploded in popularity in the short time they have existed. As was mentioned before, according to a report put out by Emogi, 92% of online users use emojis at least several times a year and over 60% use them several times a week (2015 Emogi Report, 4). As emojis became more common, linguists had taken an interest in how they functioned with language and what they were capable of communicating in a multimodal setting. Marcel Danesi was one of those linguists who took an interest in decoding emojis and he compiled his findings in the book The Semiotics of Emoji, giving insight on how emojis are used from several linguistic perspectives. Looking at emojis in a general sense, Danesi lists two ways that they can relate to the text.
3.1 Defining Adjunctive and Substitutive
In his book The Semiotics of Emoji, Marcel Danesi gives an overview of how emojis are used from a linguistic point of view, emphasizing that users must have “emoji competence”. This means that the user understands the patterns of emojis, how they relate to accompanying text, and that the user is able to execute them in a meaningful way. Further, he explains that there are two methods of using emojis according to message structure: adjunctive and substitutive (Danesi 2017, 36). Using the first method, adjunctive, emojis only complement the information that is already communicated through the text. Adjunctive texts…are constructed with a blend of written (alphabetic) forms and emoji ones located at specific points in the flow of the message. The kind of competence required to decode such a text is likely to be more accessible to anyone—emoji users or not (36). They often have a reinforcing or illustrative quality, supporting the ideas already given in text, making them easy to decode. The second function, substitutive, is when an emoji fully replaces a segment of text. In other words, removing the emoji would change a message’s denotative meaning (Danesi 2017, 36). We can see some motivation for using emojis to substitute text since their creation. One of Shigetaka Kurita’s goals was to communicate ideas through simple icons instead of words, for example 8 using emojis for weather instead of text to make mobile applications easier to use (Blagdon 2013). It may be generally acknowledged that symbols can function as a substitute for some words, but there is remarkably little information on the substitutive properties of emojis in writing, their patterns of use, and possible limitations. Danesi is not the only person to write about how emojis are used. Several publications have looked closer at the functions of emojis, as well as emoticons, and how they influence communication. Before tackling literature on emojis, it is worthwhile to review previous work on emoticons, the emoji predecessor, which function in a very similar way.
3.2 Emoticons
When speaking about existing literature on the use of emoticons, it is important to point out that there are no publications discussing emoticons as a substitute for text. However there are numerous studies focused on their adjunctive use, accompanying text instead of replacing it. One of the main aspects that literature focuses on is how emoticons can help maintain interpersonal relationships between users and guide message interpretation. Since emoticons resemble facial expressions, over the years there have been a few publications that discuss the possibility of emoticons functioning as surrogates of nonverbal facial cues in written language. Linguist David Crystal comments on the idea of emoticons as being a form of nonverbal communication in his book Language and the Internet. He points out that emoticons are not the same as facial expressions since they are a conscious addition to language, unlike the spontaneous nature of facial expressions in face-to-face conversations. Further, he says “it is plain that they are a potentially helpful but extremely crude way of capturing some of the basic features of facial expression, but their semantic role is limited.” (Crystal 2006, 37) While emoticons are not the same as facial expressions, researchers like Masahide Yuasa, Keiichi Saito, and Naoki Mukawa have considered to what degree they may function as nonverbal communication. The researchers investigated this by collecting fMRI data as subjects read sentences with and
9 without emoticons. After reviewing the collected data of brain activity, results supported the possibility that emoticons require brain processes that involve both verbal and nonverbal communication (Yuasa, Saito and Mukawa 2011). Sharon Y Tettegah and Martin Gartmeier also comment on the nonverbal effect of emoticons in their book Emoticons, Technology, Design, and Learning. They claim that emoticons “represent emotional and personality nuances present in face-to-face communication” and that they help deliver nonverbal content which gets lost in text (Tettegah and Gartmeier 2016, 177). Several studies have also looked at how emoticons maintain relationships in terms of politeness. For example, after reviewing interactions from internet blog posts, Barry Kavanagh found that emoticons were used at the end of messages as a positive politeness strategy to avoid conflicts with other users (Kavanagh 2016). Emoticons were also found to be motivated by politeness in workplace emails, primarily as hedges to soften messages that otherwise may have a harsher tone (Skovholt, Grønning and Kankaanranta 2014). A study from Sara H. Hsieh and Timmy H. Tseng also confirms that emoticons contribute playfulness and help facilitate social connectedness (Hsieh and Tseng 2017). Eli Dresner and Susan Herring took a closer look at how emoticons maintain positive interactions between users. They identify two major ways emoticons are used in terms of pragmatics: 1. As a direct mapping of expressions from face-to-face interactions. The expression may accurately reflect the user’s emotional state or be a forced expression for the sake of politeness. 2. As an illocutionary force indicator. Illocutionary force indicators (examples being punctuation, tone, volume, etc.) guide the reader or listener to the intended understanding of a sentence (Dresner and Herring 2010). For example, we use illocutionary force indicators to detect whether the sentence “Did you take my notebook?” is meant as an accusation or a genuine question. In this instance, we may use indicators like tone and facial expression. Dresner and Herring’s second observation, emoticons as illocutionary force indicators, parallels Scott Fahlman’s explanation about why he first used emoticons, mentioned in 2.1 Development of the Emoji. Fahlman explained
10 that the emoticons indicated whether a message should be read as a joke or with a serious intention. The idea of emoticons being illocutionary force indicators has been supported by several other studies, especially in the case of marking humor or sarcasm (Wolf 2000). There is also some data available on the placement of emoticons in messages. In 2011 it was found that a strong majority of emoticons appeared in the final position of a message, in place of punctuation. Only 20% were found in the middle of messages and much fewer at the beginning (Garrison, et al. 2011). There are also several studies considering how certain demographic groups used emoticons differently, like a tendency for men to use emoticons as a marker of sarcasm over women (Wolf 2000), but they will not be covered here in depth. Having a short look at the literature on emoticon use we can get a general picture of how they are used adjunctively with the text. Some believe that emoticons partially act as nonverbal cues. Other publications focus on the influence emoticons have on illocutionary force and sentence interpretation, especially in the case of politeness. Finally we also see that there are some patterns involving the placement of emoticons in a message, with results showing that they usually appear at the end. Having this in mind, we can move on to literature about the emoticon’s successor.
3.3 Emojis
Having acquainted with how emoticons function, we can move on to emojis. Emoticons hold many similarities to the face emojis, and in some cases there are separate publications about emoticons and emojis that draw the same conclusions. However, when comparing the number of journal articles on emoticons and emojis, there are far fewer examples of literature focused on emojis and much more on emoticons. This may be because of the novelty of emojis. Although there is less written about emojis, they have been the subject of a handful of studies since their standardization in 2009. Unfortunately none of the studies discuss the substitutive properties of emojis, but they do paint a picture of how they are used adjunctively to text.
11
One similarity between emoticons and emojis that we can find in publications is in expressing emotion and politeness. After conducting interviews between 2006 and 2010 in Japan, Satomi Sugiyama found that in addition to using emojis for emotional expression, they were also used to maintain pleasant relationships and politeness, making messages more friendly and positive (Sugiyama 2015). Using emojis for maintaining pleasant relationships is also discussed by Marcel Danesi, who says they “make a message positive in tone and overall effect” and that they act like “emotive punctuation marks” at the end of messages (2017, 79). Danesi considers this to be one of the pragmatic functions of emojis, injecting a positive mood. The other function he lists is adding tone, explaining that emojis provide “a visual means to convey prosodic meaning” (2017, 95). The emojis then help guide the reader to the intended meaning, acting as an illocutionary force indicator, just like Dresner and Herring’s findings with emoticons. Up to this point the literature that has been reviewed is primarily based on data of emoticons and face emojis. The face emojis only make up a small part of the whole emoji inventory, but fortunately there is one study that collected data on the performance of all emojis. More specifically, the study records the placement of emojis in messages as well as their sentiment. Petra Kralj Novak, Jasmina Smailović, Borut Sluban, and Igor Mozetič’s collect data on emoji sentiment and placement from over 1.6 million tweets in 13 European languages. This data, which looks at the behavior of all emojis, can give us some clues on whether there is a difference in how the face emojis and the other emojis are used. In terms of placement, the researchers find that in general emojis tend to be placed at the end of messages. Taking a closer look, the researchers notice a correlation between sentiment and placement. Patterns show that the closer emojis were placed to the end of a message, the higher sentiment value it expressed, whether positive or negative (Novak, et al. 2015). This can be seen on the graph of emoji data in Figure 3 below. Looking more closely at their results, we can see that the face emojis performed differently than the other emojis.
12
Figure 3: Graph of data from Novak et al. The x-axis measures placement, 0 being the beginning of a message and 1 the end. The y-axis measures neutrality. The lowest score, 0, means the emoji was not used with neutral sentiments at all, only positive or negative. The highest score, 1, means all instances of the emoji held neutral sentiment
Looking at the data collected from Twitter messages, the face emojis (seen as a cluster of yellow in the bottom right) consistently scored with more extreme sentiment values and more final placement. However the other emojis, like object and symbol emojis, are peppered across the middle of the chart, with a cluster to the upper left of the face emojis. This meant that they tended to have less extreme sentiment rankings and had an average placement more central to a message. The different performance of face emojis may be because they were used primarily as an adjunct to text. As was mentioned earlier in this section, adjunctive emojis tend to appear last because they are often used like emotive punctuation, guiding the reader’s interpretation of a message as an illocutionary force indicator. Danesi and others have also said that the face emojis are used as mood indicators, most often to express positive emotion. This expressivity correlates with the high sentiment scores of face emojis. The other emojis outside of the face category had an average placement closer to the center of a message. This may be because they were not used adjunctively to text, like the face emojis were at the end of messages, but to
13 substitute a word. As a substitutive emoji they would also primarily have a denotative meaning, not expressing emotion, so their sentiment score would likely be less extreme than the adjunctive face emojis. This data possibly shows that the face emojis were used in a different way than the other emojis, potentially due to the difference in adjunctive and substitutive use. Unfortunately there is no published data on substitutive emojis or much literature on the use of non-face emojis to corroborate this trend in the above data.
3.4 Identifying the Gaps in Knowledge
After reviewing the available literature about emojis as a means of communication I have found that, while there are several studies on the adjunctive use of emojis, there is little to no data regarding emojis as a substitute for words. Marcel Danesi acknowledges the possibility of substitutive emojis in his book and there are several prominent examples of emojis replacing text (which will be discussed in-depth later), but there is no tangible data that considers how substitutive emojis can be used in a practical sense. While reviewing studies on emojis I also found a strong focus on the face emojis, which only make up a small percentage of the emoji inventory, and few studies on any of the other emojis. Results from Novak, Smailović, Sluban, and Mozetič’s research on emoji placement and sentiment potentially show evidence that the face emojis are used more often adjunctively while the other emojis are used substitutively. However this cannot be determined because there is a lack of data on substitutive emojis and the performance of non-face emojis. These gaps of information are what this thesis will be focusing on. Before collecting data to fill in these gaps, I would like to review similar phenomena and examples of substitutive emojis thus far. This will help with understanding the possible capabilities and limitations of substitutive emojis and give clues on how they may behave when paired with text. This knowledge will then help with deciding which data would be most helpful to collect to better understand substitutive emojis.
14
4 Images Substituting Text
Until this point we have looked at some of the adjunctive possibilities of emojis discussed in literature. There is an incredibly small amount of research about emojis from the perspective of linguistic substitution, yet there have been instances of messages, and even entire novels, made up of only emojis. Since there is little written about emojis substituting language, similar phenomena will be discussed. More specifically pictographs and rebus writing will be examined. Looking at these alternative methods of conveying information can help with understanding the limits of graphic-based communication and what potential emojis have in substituting words in written language.
4.1 Pictographs
Long before emojis were first made, many early methods of writing involved pictographs, which are symbols representing objects and ideas (Ottenheimer 2008, 185). However for a symbol to be considered a pictograph, it must visually resemble what it denotes. The majority of emojis depict concrete ideas, like objects and people, making them pictographs. This is contrary to ideograms, which are symbols that do not directly resemble their denotation (Yule 2010, 212). An example of this would be a hazardous waste symbol. There are several benefits to pictographs that our standard phonographic language today does not have. Pictographs can be a quick, effective form of communication and comprehension does not rely on knowledge of a particular language, so the symbols can be used internationally (Yule, 213). Pictographs are common even today for their quick speed in communicating information, like in the case of road signs. They are indeed used in some international contexts since no language system is needed to understand them. Examples are map symbols and some signs in prominent institutions, like the do not smoke symbol. Pictographs are also often used today as a tool for individuals who have trouble communicating using traditional methods. A sample page of a medical visual language translator can be found below, a communication tool. Specifically, the sample page is meant to be used for situations when someone
15 is injured from a bite. Using the translator, a person can point to the appropriate picture to easily communicate the circumstances of the injury.
Figure 4: A page from a medical visual language translator (Stillman 2013)
While pictographs can be helpful in some situations, they have some significant drawbacks as well. Because pictographs heavily rely on visuals for their denotation, the symbols are limited to representing concrete forms. Complex abstract ideas, like consumption, are difficult to express through a pictograph unless they are conventionalized, like the symbol used for recycling. In addition to that, pictographs also do not convey grammatical information, like tense and person. This limitation means that pictographs cannot form a complete writing system on their own (Ottenheimer, 185). Relating this back to the thesis topic, emojis can be considered pictographs to a large degree since they depict concrete ideas like facial expressions and objects, an exception being the symbol emojis which include common symbols for more complex ideas, like religion and the zodiac.
16
Because emojis focus on depicting concrete ideas, they are easy to understand by people around the world, just like pictographs. Danesi makes a comment to reinforce this idea, saying emojis have “…enough universal properties built implicitly into it, given its picture-word nature, to make it much more amenable to interpretation than alphabet-based texts” (37). The emojis also attract attention and deliver information in a different way that plain text cannot with their colorful graphics. As was mentioned earlier, they make the communication more fun. Danesi does also mention a few differences between historical pictographs and emojis. One difference is that emojis are much easier to reproduce and each emoji has a consistent look. The only variation is in gender, skin tone, and hair modifiers. On the other hand past historical pictographic symbols were hand drawn, giving more inconsistency and individuality to the constructed forms (52). When thinking about the limitations of both emojis and pictographs, we see that that there is overlap. The pictographic nature of emojis makes it difficult to communicate more complex ideas, especially in the case of expressing grammar and more abstract parts of speech. As pictographs, emojis are largely limited to expressing grammar through ordering and placement. However to express grammatical meaning there is another form of picture- writing that emoji users can consider called rebus writing.
4.2 Rebus
Emojis have limitations because of their pictographic nature, but rebus writing is one method of communication that can bypass those limitations to a degree. Rebus writing is when a picture is used which can stand in for two or more words that sound the same (Ottenheimer 2008, 186). Another way of putting it, instead of writing a word with text, a picture of an object is used that is its homophone. For example, in rebus writing if a person is trying to communicate the pronoun I, a symbol depicting a human eye can be used because eye and I are the same phonetically. When a series of these symbols are placed adjacent to one another, they can form more complex phrases, like the series pictured in Figure 5 below. Using this rebus method, the emojis can be read as the
17 sentence I can see you. The corresponding homophones would be eye-can-sea- ewe.
Figure 5: An example of rebus writing using the eye, canned food, wave, and ewe emojis
The benefit of using rebus writing is that more complex ideas can be communicated along with its grammatical aspects. From the placement we can already identify which are most likely to be the subject, verb, and object. This is also possible with pictographs. Here the rebus-emoji phrase can also communicate the subject’s and object’s number and person. From that the reader can deduce what the form of the verbs should be. In some cases where it would be difficult to use a pictograph for a word, like the pronoun I, the rebus method of writing may be the solution. However rebus writings do take extra effort to read. The reader must recognize the object meant to be depicted and then process the pronunciation of the object, mapping it to a homophone before understanding what is being communicated. In fact, their difficulty in understanding is why they are the basis of several puzzle activities meant to challenge the reader. Danesi comments on their difficulty as well, saying that large amounts of emojis make texts complicated with “visual noise” that a person must wade through and process before discerning any meaning. Users he interviewed reported that emojis were difficult to read while in rebus form, but interesting (2017, 90-91). Although many people agree that these kinds of emoji-heavy texts, like rebus-emoji phrases, are difficult for comprehension, it has not stopped some individuals from trying to communicate through emojis alone.
4.3 Emoji-Only Communication
Since the popularization of emojis, several examples of emoji-only messages and prose have emerged. In one case a narrative made of emojis was even published as a book. Using emojis to convey such complex information goes against the nature of pictographs which, as was discussed before, can only communicate simple concrete ideas successfully.
18
When discussing pictographs it was mentioned that emojis on their own are not able to express grammatical information except through placement. Rebus writing provides an alternative to express this information, but is decidedly difficult to decode, especially longer passages. When it comes to texts that are composed of much longer stretches of emojis, Danesi comments that there may be a different type of organization and structure to deliver meaning which does not rely on the syntactic rules of English at all: Emoji grammar is often nothing more than a “placement grammar,” based on calquing, or the superimposition of emoji in slots where verbal structures would otherwise have occurred if the text were written entirely in words. However, in some texts, there is a different system at work, whereby the sequencing and compositional aspects are governed by conceptual aspects, rather than strict rules of grammar. In such cases it is obvious that there is a pictorial-conceptual grammar involved….The point is that emoji grammar is not just a replica of linguistic grammar with visual symbols; it has its own “syntactics,” or system for organizing the emoji to create coherent and meaningful sequences or combinations. (2017, 77-78)
Danesi here describes a different kind of emoji composition; one that reads as a general conceptual illustration of an idea or situation. With these different types of emoji structuring in mind, we can have a look at some examples of emoji-only communication and consider how successful they are at communication despite their pictographic limitations. One interesting resource for shorter emoji-only phrases is emojisaurus.com. The website launched on the 19th of October 2013 and collects emoji translations of English phrases. Registered users are given an English phrase and must attempt to translate them into emojis, called emojigrams by the website. The emoji translations vary in legibility, with some being incredibly difficult to grasp. In Figure 6 there is a translation of a known phrase, early to bed, and early to rise makes a man healthy, wealthy and wise. Looking at the emoji translation we can see that the calquing approach was used since the emojis correspond to each word of the original phrase. Early here is shown as the
19 sunset ( ) and 7 o’clock ( ) emojis to represent the concept of an evening hour, which could be considered early for sleep. It does not directly convey the concept of early, but it does communicate a similar idea. The sleeping face emoji ( ) logically represents the person in bed, replacing to bed. Following that, the plus emoji (➕) is used in place of and. The rest of the translation continues with this looser unit-by-unit approach to translating the phrase.
Figure 6: One example of an English phrase, Early to bed and early to rise makes a man healthy, wealthy and wise, translated into emoji from Emojisaurus.com (Emojisaurus)
The translation shown in Figure 7 uses rebus writing, which was discussed in the previous section, to show Mount Kilimanjaro as a series of emojis. The emojis in the translation correspond to the pronunciation of the original phrase, particularly with the name of the mountain Kilimanjaro. The man and gun emojis ( , ) are meant to be read as kill a man which sounds similar to the first three syllables of the name. The last two syllables are shown as the honey pot and red circle emoji ( , ⭕), meant to be read as jar and O respectively. Together they phonetically sound out as something very close to Kilimanjaro.
Figure 7: An emoji translation of the phrase Mount Kilimanjaro using rebus writing (Emojisaurus)
Not all the translations obeyed the English language structure so carefully. In Figure 8 below we can see that instead of the translation going
20 from unit-to-unit, the emojis illustrate a whole situation. The beginning shows a woman rejecting something, perhaps a request from another person to do a task, and the emojis following show all the activities the person is busy with, including applying makeup, getting dressed, and having coffee. At the end of the translation there are three emojis that give extra information, that the person is possibly busy getting ready for school. This information is not given by the original phrase, so the translation is not entirely accurate. Inconsistencies aside, this sequence of emojis is constructed not according to the English language, but on a conceptual basis. The emojis illustrate the situation and circumstances of a phrase and comes across as a collage of ideas.
Figure 8: Another Emoji translation, from the phrase I’m busy getting ready (Emojisaurus)
The three emoji translations above were all executed differently, but all of them are difficult to translate back into English. While one might be easier to understand than another, reading a large block of text in any of these styles of translation would be taxing on the reader. Given a choice between the original English phrase and the emoji version, the text would be a much more efficient and convenient way to relay that information. The examples above were only short translations made by average users online, but there are also examples of emoji-only communication from prominent institutions. Chevrolet published an advertisement in all emoji for one of their new cars in 2016. When Marcel Danesi showed the advertisement to a test group “virtually all of them expressed difficulty in translating it” and several members of the group suggested that it may only be to communicate an overall concept, and not an organized complex idea which is normally conveyed through sentences (Danesi 2017, 150). Danesi also mentions the publication of Emoji Dick, which is a translation of the entire novel Moby Dick into emojis, published in 2009 (Danesi 2017, 146). The novel was translated by users hired through Amazon
21
Mechanical Turk and each emoji translation is juxtaposed with the original text, showing how closely, or not, a translation follows the original novel (Melville 2010). Some sample translations can be found below. As may be expected, it is difficult to discern the intended meaning through the emojis alone.
Figure 9: Sample translations of Emoji Dick, compiled by Fred Benenson (Benenson 2009)
Looking at all the presented examples, trying to communicate messages using only emojis is troublesome, at best. Emojis have none of the grammatical systems in place needed to communicate larger organized thoughts that a language would require and trying to express them through a method like rebus writing is challenging. Presenting the ideas more conceptually with emojis presents another option, but it generally is not as informative and can be interpreted variously by users since there is no standardized emoji language system. Danesi himself comments that while there has been an upsurge in experimental emoji-only texts, he does not believe that it can develop into a regular writing style because of all the “visual noise” a reader must process and its limitations in expression (2017, 148). To use emojis substitutively, perhaps a less drastic approach must be taken for it to be legible and more articulate. Instead of going so far as to replace stretches of text with emojis, maybe there can be a solution found in combining the mediums, making a mix of word and emoji to deliver information.
22
4.4 Mixed Textuality: A Multimodal Approach
Over the last few pages we have seen that while emojis may be capable of expressing ideas, there are limits to how complex the ideas can be. In addition, the effort needed to encode and decode longer stretches of emojis do not make them practical. There is one other possibility of using emojis substitutively, and that is by using them in combination with text. This makes the communication multimodal and both modes, text and emoji, are able to work together. This way what one mode lacks the other can supplement. For example, it has been shown that emojis have difficulties in expressing grammar. Taking this multimodal approach, text can be added in alongside emojis to express them instead. The idea of using substitutive emojis and text together is only briefly mentioned by Danesi in his book The Semiotics of Emoji in one example, who instead focuses the book on adjunctive emojis. He calls the multimodal combination of substitutive emojis and text mixed textuality, “whereby the written text is used in tandem with emoji that… are designed to substitute content in specific ways” (2017, 38). The example he uses to illustrate this mixed textuality is seen below in Figure 10. The message uses text and emojis to recite the theme song of television show The Fresh Prince of Bel-Air.
Figure 10: An example of mixed textuality provided by Marcel Danesi in The Semiotics of Emoji. It reads as “Now this is a story all about how my life got flipped, turned upside down, now I’d like to take a minute, just sit right there, I’ll tell ya how I became a prince of a town called Bel-Air” (2017, 38)
When used in combination with text the possibilities of substitutive emojis expands considerably. The text is able to communicate what the emojis
23 cannot and the interweaving of text provides a “break” from decoding emojis. The text still adheres to the English language system, only using emojis to replace decipherable parts, making the text more colorful and playful.
4.5 The Plausibility of Substitutive Emojis
In the above sections I reviewed two forms of script which closely resemble substitutive emojis, more specifically pictographs and rebus writing. After comparison, I determined that emojis and pictographs do indeed have a strong resemblance. Just like with pictographs, emojis are largely limited to expressing only what they are able to depict, concrete entities. Using emojis in rebus form helps to expand those limitations to a degree, but the complications in encoding and decoding rebus communication makes it incredibly impractical as a means of communication. Then some emoji-only messages and passages were reviewed. Marcel Danesi suggests that these emoji-heavy texts have potential to express more general concepts, illustrating situations as a collage of images. However there is no universal system in place to use emojis in this way, making them difficult to decipher and interpretations inconsistent. It would seem that emojis have great difficulties in substituting text on their own. This is not entirely surprising. The two modes have their own, entirely different methods of relaying information and trying to use one mode strictly within the system of another is like trying to fit a round peg in square hole. The most practical solution seems to be the multimodal “mixed textuality” approach, combining text and substitutive emojis. This approach would allow for the modes to complement each other so that the benefits of each are present. This multimodal approach would allow text to supplement the information that emojis cannot express so easily, like grammatical information and abstract ideas. Unfortunately, there is no published data on using substitutive emojis to determine how they behave. For the rest of this thesis, I would like to fill in this gap of information.
24
4.6 Hypotheses to Test
After reviewing current literature on emojis, we can see that there is a large gap in knowledge. While there are studies on how emojis work with language, none of them are focused on their substitutive properties, only their adjunctive ones. Substitution is a function of emojis that is acknowledged by Marcel Danesi and examples exist, but without any tangible data it is impossible to determine how substitute emojis are used in communication. After reviewing similar phenomena, the limits of substitutive emojis can be predicted. Substitutive emojis are small static images, so they can largely only communicate simple concrete ideas successfully. Because of this they are likely most effective when used in conjunction with text, creating a multimodal text. What I would like to collect data on is which part of speech substitutive emojis replace most. Hypothesis 1: Nouns are expected to be replaced the most by emojis. This prediction is made because substitutive emojis are limited to depicting simple concrete ideas. Nouns are the most closely associated with expressing these ideas. The other word classes tend to represent more abstract or complex ideas that are difficult to visualize as a pictogram. In addition, nouns have minimal grammatical affixes to complicate substitution. Analyzing the samples of substitutive emojis collected for this data will then show how users handle the translation of. Many words have grammatical information encoded in their forms through inflectional endings. If words are simply swapped by emojis, the information given through these grammatical endings are lost. Looking at the examples more carefully will reveal how big of an obstacle this was in substitution and how users overcame these complications. While reviewing current literature I also noticed there was only a small range of emojis studied. Almost all the data from current studies on emojis concern themselves with the face emojis, which only make up a small part of the 1,262 existing emojis. The studies focused on these emojis confirm that they do indeed have strong adjunctive functions. They especially point out the emotive effect of face emojis, a non-verbal aspect of language.
25
A study by Novak, Smailović, Sluban, and Mozetič shows evidence that the face emojis perform differently with text than the other emojis. The data suggests that the face emojis are primarily used adjunctively and the others substitutively. Because current studies only look at the use of emojis adjunctively, there is no data to corroborate this. To remedy this, I would like to collect data on which emoji categories are used in substitutions to confirm or deny that certain emojis have a different type of relationship with text than others. Hypothesis 2: Face emojis will be used markedly less in substitutions than other emoji categories. This is because the face emojis are better suited to resemble paralanguage and non-verbal aspects of language, which only have adjunctive effects on the text.
26
5 Substitutive Emojis on Instagram
In the theoretical part of this thesis I reviewed currently available literature on the use of emojis. While there are a handful of publications discussing the adjunctive possibilities of emojis, there is a lack of knowledge about substitutive emojis. There is also little data on emojis outside of the face category. The goal of this part of the thesis is to collect instances of substitutive emojis in practice on social media and then analyze the results to find patterns in their usage. More specifically, I will be looking at which parts of speech are replaced as well as which categories of emojis are used as a substitute.
5.1 Methodology
Marcel Danesi mentions that possibility of substitutive emojis in his book The Semiotics of Emoji, but there is a lack of in-depth analysis regarding what parts of language are replaced. In order to fill in that information, first I had to compile a corpus of substitutive emojis for analysis. The social media platform Instagram was used for collecting data because of its public availability and the fact that it houses the largest active userbase compared to similar social media with the exception of Facebook (Smith and Anserson 2018). Instagram is a social media platform that launched in October of 2010 by co-founders Kevin Systrom and Mike Krieger (Instagram Info Center 2010). The platform is focused on sharing picture content and each image post has the option of attaching a text caption. These captions are where substitutive emojis were found for the corpus. In Figure 11 below is a screenshot showing the basic format of an Instagram post for a better idea of how the caption is juxtaposed with the image content. In the green square at the top is the username of the person that created the post. In this case the name of the user is asubtlerevelry, one of the accounts that was chosen for this compiled corpus. The purple square is where the image content is placed. The image is always visible and significantly larger than the other parts of the post, drawing the most attention. This is Instagram’s way of putting a bigger emphasis on the picture content of a post. The yellow square is the most important for this corpus and is where the image caption is placed. Below that in the blue square are comments from other users and finally in the red square is the date that the post was published. 27
Figure 11: The format of Instagram posts
Some users make posts with more emphasis on the picture content, leaving simpler, short captions. In other cases there may be a bigger emphasis on the caption, making it more informative than the image. An example of this is a photograph of a woman working out and a caption outlining a specific routine over several lines of text in the caption. I compiled examples of substitutive emojis from 39 Instagram accounts. The accounts were chosen because the users added emojis in many of their captions, showing that they were comfortable and confident with using them. A variety of types of blogs were selected to ensure a wider range of vocabulary would be discussed in captions. This was to reduce the chance of the same topics and substitutions to be used repeatedly. The captions containing substitutive emojis published between the 1st of January 2018 and the 5th of September 2019 were recorded in full. In addition to that, the post’s date of publication, post identification code (found through the URL of the post), replaced part of speech, and emoji were also recorded. Each Instagram account was given its own individual document file containing a chart with the above data listed. After collecting the data is a review of the results and an analysis. Information regarding part of speech and emoji category will be analyzed to confirm or deny the hypotheses and consider why. Following that are some brief observations about trends found in the corpus regarding substitution. 28
5.2 Results
After reviewing the Instagram posts of 39 users, there were 902 instances of words being replaced by one or more emojis. In total there were 968 emojis used across all substitutions, a higher number since some words were replaced with several emojis. A range of blog types were selected to ensure a variety of vocabulary would be used. In many cases the Instagram accounts doubled as personal blogs, but some examples of the types of blogs analyzed were pet blogs, high fashion blogs, do-it-yourself blogs, photography blogs, and blogs focused on motherhood. The variety of emojis used in substitution was quite wide, with 304 unique emojis. The number of times a single emoji was used ranged from 1 to 57. Most emojis however belonged to the lower end on the range of frequency. There were 165 emojis which were used in substitutions once and 67 emojis used two times. Only 48 emojis were used 5 or more times. Results concerning the replaced part of speech can be seen in Figure 12 on page 30. Emojis replaced nouns at a much higher rate than any other part of speech. More specifically, nouns were replaced in 69% of substitutions. The next most common part of speech was verbs, being replaced in a significantly lower number of cases at 11%. Other parts of speech were substituted even less often, with adjectives substituted in 9% of examples, adverbs in 7%, and the remaining parts of speech like prepositions, pronouns, and numerals were collectively replaced in 4% of substitutions. Nouns were replaced far more than any other type of speech, over six times more than the next largest category, verbs. This confirms the hypothesis that nouns would be the most frequently replaced part of speech. Verbs were replaced with the second highest frequency at 11%. One reason why they scored more than other categories may simply be attributed to the structure of the English language. Verbs (and nouns) are necessary for a complete sentence while the other lexical word classes (adjectives and adverbs) are not. This would mean that verbs appeared more often than adjectives and adverbs, getting more opportunities for substitution. However adjectives and adverbs were substituted only slightly less frequently than verbs, amounting to 9% and 7% of substitutions respectively.
29
P Figure 12: Pie chart showing the parts of speech emojis replaced in Instagram captions
The “other” category contained prepositions, pronoun, and numerals. These parts of speech may have appeared less because they do not refer to specific entities that could be visualized uniformly, perhaps except for numerals. For example the idea of I and you are relative to the communication and the referent changes with context. However, there were examples of these word classes being substituted, and this will be reviewed in more detail in the following analysis section as well. After reviewing the numbers for parts of speech, now we can look at the selection of emojis. In my hypothesis I predict that face emojis would be used considerably less than other categories. Previous studies show that the face emojis more accurately resemble paralinguistic and nonverbal aspects of communication, playing a large role in sentence interpretation and politeness. This means that they would primarily be used adjunctively to text and not contribute denotative meaning to a message as substitutive emojis do. Below is a pie chart showing the number of emojis used in each category. These totals count each emoji used in substitution. If several emojis replaced a single word (example for travel), each emoji was counted individually. However if the same emoji was repeated when substituting a single word (one example being for sun), this counted as a single instance of that emoji.
30
N Figure 13: Pie chart showing which emojis were used in substitutions
Looking at Figure 13 above, the results of the emoji categories (including repetitions) were spread out more evenly than the part of speech data. In total 968 individual emojis were used, a higher number than the number of substituted words since in a few cases several emojis replaced a single word. The leading category by a modest margin was food & drink at 22% (210 emojis). The next four categories all had a similar number of emojis used: objects with 15% (149 emojis), people & body 15% (145 emojis), emotion also at 15% (142 emojis), and travel & places slightly less at 14% (134 emojis). The animals & nature category was next with 11% of emojis (102 emojis), and the remaining categories (symbols, face, and activities) had percentages in the low single digits. Looking at the number of unique emojis used in Figure 14 below we can see the breadth of emojis used from each category. These numbers eliminate the influence of emojis with extraordinarily high frequency, like the heart emoji, on category ranking.
31
Figure 14: A pie chart showing the numberN of unique emojis used per category
The categories performed differently when looking at the data from this perspective. The strongest similarity is with the leading two categories, food & drink (22%, 67 unique emojis) and objects (16%, 49 unique emojis). The biggest difference between the two graphs involves the emotion category. While there were relatively few unique emojis used from this category (19), certain emotion emojis were used so frequently that the category placed much higher in overall use (142). This category had the most used emoji (the heart emoji , 57 substitutions) as well as the second most used emoji (100 emoji , 37 substitutions). As expected, both methods of interpreting data show that categories featuring concrete entities were most popular for substitution. The top 3 categories (food & drink, objects, and people & body) all feature physical objects or individuals. This follows the idea that pictographs are most successful in depicting concrete entities when used as a part of writing. However in order to confirm or reject the hypothesis we must focus on the face category. Looking at the data from either perspective, face emojis were used sparingly. Only 22 of the 968 emojis found in substitutions were from the 32 face category (2%). When the repetitions are removed from data, face emojis only appeared 11 times (4% of all unique emojis). This confirms the hypothesis that the face emojis would be used among the least for substitutive use. The following analysis section will have a more careful investigation of which face emojis were used and how, as well as consider how some of the other categories were used.
33
6 Analysis
After having a short overview of the data collected, I will take a closer look at each part of speech to consider why certain parts of speech were substituted more and others less. To do this, some of the favorable qualities and possible obstacles that each word class presents for substitution will be considered. Examples from the corpus will also be presented to show how users handled the complications in practice. This part of the analysis focuses on the inflectional endings of each word class since these pose a challenge for pictographic substitution. As was mentioned earlier, emojis have difficulty communicating aspects of grammar given through inflectional endings, like gradation and tense. With some word classes the users are able to express the grammatical information in a variety of other ways or the reader could infer it through context. With other parts of speech the substitution was more problematic, making the caption difficult to understand. After this, emoji selection will be examined. Looking at the examples more carefully may reveal why certain emojis were used more frequently than others. This part of the analysis will be done by emoji category. Afterwards are some additional observations about the possible motivations users had in using emojis instead of words, message composition, and some unique substitutions. When discussing examples from the compiled corpus in this analysis, many captions are partially quoted for the reader’s convenience. In places where it was more appropriate to show the entire Instagram post, including the image content, a screenshot of the post was made and included as a figure at the end of the thesis in 9 Examples on page 67.
6.1 Parts of Speech
Because pictographs convey information differently than words, substitution was not always a simple act of swapping a word for an emoji. In this part of the analysis the potential challenges that each word class poses with substitution will be discussed, especially inflectional endings expressing grammatical information. Specific examples will also be presented to show the different ways users handled these obstacles. 34
In almost all cases the part of speech was easy to identify. This is probably because of the nature of the English language, which has a grammaticalized word order. This means that the part of speech is easy to identify through the word’s placement in a text. Fortunately, emojis are able to express that same information through placement as words do. In other languages where the word order is less strict, like Czech, the word class of the emojis would be more difficult to determine through placement alone. The most commonly replaced word classes were nouns, verbs, adjectives, and adverbs. There were also a few prepositions, pronouns, as well as numerals substituted by emojis. These were not expected to be substituted since they express more complex concepts involving grammar, which could be too difficult to communicate through emojis. Aside from these word classes, there were some more unique substitutions that were not expected. In one case there was an entire expression replaced by an emoji. This substitution and other irregular substitutions will be looked at more carefully later in the analysis.
6.1.1 Nouns
As was mentioned in the results section, nouns made up 69% of substitutions. This high frequency is not surprising since pictographs can only communicate what they can visually depict, and nouns concern concrete objects more often than any other word classes. From here we can discuss the difficulties nouns present for substitution and then take a closer look at examples from the corpus to see how users handled those complications. English nouns have few inflectional endings. Their simple grammatical form could be another reason why they were replaced so frequently. There are four possible forms for nouns and examples of them are listed below in Table 2, borrowed from The Cambridge Grammar of the English Language (Huddleston and Pullum 2002, 326). The forms of dog represent regular endings and the forms of child irregular endings.
PLAIN GENITIVE PLAIN GENITIVE SINGULAR dog dog’s child child’s PLURAL dogs dogs’ children children’s
Table 2: The different forms of English nouns
35
The examples above show -s, –‘s, and -s’ as possible regular inflectional endings for nouns. These endings express whether a noun is plural or in the genitive case (expressing possession), respectively. If a noun form were simply swapped with an emoji, these endings and the information they communicate would be lost. However, Instagram users in the corpus found a few ways to handle the loss of information or found a different way of expressing these features. First, we will discuss the various ways users expressed the number of substituted nouns. One method used by some bloggers to communicate the plural endings was to simply add the -s ending after the emoji. This way the emoji stood in for the stem and the inflectional ending -s followed directly after. This combination lets the emoji grab the reader’s attention and makes the caption more fun with a colorful image while the -s ending easily clarifies the number. This can be seen in one of dollyandoateal’s captions describing a recipe below in (1a). The same method can be seen in (1b) from desiperkins, who refers to her followers as chicks. (1) (a) All you need are some russet s, a handful of shallots, some soaked cashews, almond milk, and some pantry goods for this (#vegan) Hasselback Potato & Shallot Gratin. (b) How are all my ’s today? The apostrophe with the “s” in (1b) may cause some confusion since it looks like a possessive form, but this would be an awkward way of forming the question, so it is likely meant to mark that an abbreviation was made. Aside from examples where the user added an apostrophe, the -s ending tacked onto an emoji is a clear sign of the noun’s number. Interestingly, desiperkins substituted the same word in another post but used a different method to express number. (2) Where my NYC at??? In (2) the emoji is repeated to show that the noun is plural visually instead of relying on traditional language. As was mentioned, adding emojis to the captions makes the text multimodal, using both text and image to communicate a message. This method of showing number takes advantage of the visual side of the multimodality. Another example of this can be seen in Figure 15, where
36 the user asubtlerevelry uses several butterfly emojis ( ) to express number visually. In some cases the -s endings were not given and the emoji was not multiplied to express number visually. Despite this, the number of a noun could still be determined through grammatical agreement with other words in the message. In many cases a determiner was placed before the emoji to signal whether a noun was singular or plural. One example from user brahmino seen in Figure 16 uses the determiner a. Grammatical agreement dictates that the singular form of a noun follows. The image content of the post supports this as well, showing a single goat-shaped cloud. The number of a noun could not always be identified through textual context but in many cases the surrounding text made the number of the noun easy to identify. This means that in some cases omitting the -s ending was acceptable as the information was communicated elsewhere in the sentence. In other cases the number was not signaled through the caption but could easily be implied. One example is found in Figure 17. In the post the user describes an in-home spa day and it can be implied that the bathtub emoji ( ) replaces a word like tub in the singular form. This is because generally a person will only have one personal tub in their home. The same can be said about the caption in (3). The writer uses a single emoji to replace a word and the number cannot be determined through grammatical agreement with other words nor the image content. However because it replaces a word in a common phrase (“thanking my lucky stars”), the reader can determine that the replaced word was plural. (3) Thanking my lucky my grandmother is in town to help me with kiddos today! With some captions the reader could not discern the number of the noun from the text alone. Because these examples were collected on Instagram, the users could refer to the image content of the post for clues to determine number. This can be seen in Figure 18, where it is not certain if the emoji refers to girl or girls through textual context. Looking at the image and seeing one woman standing by the palm trees, the reader then can determine that the replaced word was singular. Even in cases where the number could not be
37 determined through the image, it did not significantly hinder the understanding of the caption and the reader could still easily comprehend what was being said. While the number of nouns could be expressed or determined in several different ways, there were no substitutions involving the genitive form. Perhaps it is because the form is less common or because proper nouns cannot be expressed through emojis, making possessives like “Ryan’s” incredibly difficult to write with an emoji. Aside from inflectional endings, nouns do also take on several derivational endings like -er and -tion. There was one notable example involving a derivational ending. The substitution can be seen in (4) and Figure 19 shows the post in full with the image content. In the caption the user comments on a photography series she does on Instagram where she reimagines an outfit of a celebrity for a woman of her stature.
(4) Getting harder to #supersizethelook Meghan Markle’s looks bc she is limiting appearances before the , but this green dress/blazer combo was a no ’er.
This substitution uses the brain emoji ( ) for the stem of the original noun and then adds the derivational ending -er. Together they read as a whole word, brainer. The substitution is part of the fixed phrase “no brainer” (meaning something is obvious and does not need much thinking) making it easier to identify as well. Combining forms like this expands the breadth of emoji substitutions for nouns considerably. Another possible example could be er for heartbreaker. However with only one example found in the data, perhaps there is too much effort needed to compose and decipher these derivational combinations for it to be practical. Nouns were the most frequently replaced part of speech in the compiled corpus, and after looking more closely at the captions we can see evidence of why that is. Aside from the fact that they are easy to visualize as pictographs, they also have few inflectional endings. The inflectional endings that nouns do have did not pose a big challenge in substitution. The plural form was easily substituted by emojis. If it was not expressed by the substitution itself, the
38 agreement with surrounding words was able to help denote what number the noun should be. Even when this was not the case, the number could often be assumed through the context of the caption. There may be some complications in translating nouns in the genitive form because there were no examples of these substitutions in the corpus. The lack of these examples may even signal that these kinds of substitutions would require too much effort to encode and decode to make the substitution worthwhile.
6.1.2 Verbs
Moving on from nouns, the next most common part of speech was verbs with 100 substitutions, making up 11% of all substitutions. One possible reason why verbs were replaced radically less than nouns is because of the graphical limitations of emojis. Emojis only have still images. Many verbs may be illustrated successfully with a snapshot of the verb in action, but more dynamic verbs like jump would certainly be illustrated more clearly with a moving image. Verbs also have a much more complicated grammatical structure than nouns which may pose a challenge for substitution. Before going forwards, it may be appropriate to distinguish lexical verbs from auxiliary verbs. Auxiliary verbs are “a closed set of verbs marking meanings associated with aspect, voice, or modality” (Biber, Conrad and Leech 2002, 455). This includes the modals can, may, will, shall, must, ought, need, and dare as well as non-modals be, have, and do (Huddleston and Pullum 2002, 92). All other verbs are considered lexical and act as main verbs in clauses (Biber, Conrad and Leech 2002, 458). Looking through the substituted verbs in the corpus, only lexical verbs were replaced, no auxiliary verbs. The emoji substitutions draw a lot of visual attention. Because of that it would make more sense to replace the main verbs since auxiliary verbs only have a minor role and do not express propositional meaning. Lexical verbs have the option of several inflectional endings to complicate substitution, which can be seen in Table 3 borrowed from Huddleston and Pullum below (2002, 74). Another possible reason why verbs were less frequent is because their word forms use inflectional endings to express grammatical person, tense, and aspect. Looking at the paradigms, lexical verbs are able to take on -ed, -s, and -ing inflectional endings. The
39 preterite form could pose a bigger challenge since it causes some verbs to change their stems, as is illustrated by the take column.
Table 3: The different forms of lexical verbs Despite the complication, readers could still imply what the original form of the verb was in a few ways. In many cases the grammatical endings of the verbs were omitted entirely through substitution, but the surrounding wordforms gave readers enough context to fill in that missing information. This is an advantage that progressive verbs had. Progressive verbs in require auxiliaries which change their form according to tense and aspect just like the substituted lexical verb. Because of this, the form of the auxiliary verb could be used as a clue to fill in the grammatical information that was lost when substituting the lexical verb. In (5) below we can see the movie camera emoji ( ) substitutes the word filming or a similar verb. The preceding was signals that the emoji replaces a verb in the past progressive. This means that the correct form of the replaced verb would have the progressive -ing ending. (5) In attempt to get a hair in the air shot @dcfoodporn was firing away and @mr.cheatdayeats was the whole thing. The same advantages can be seen with the perfect aspect, although there was only one example of a perfective substitution found in the collected data. The perfective verb is found in (6) from user hannahbronfman. The words have you give us information about the aspect and person, which is enough to determine the form of the whole verb phrase. (6) Have you ever from street food When auxiliary verbs are provided, the replaced verb form can be assumed through agreement. This is true with the progressive and perfect
40 aspects. The simple aspect however does not have this advantage and differentiating from the present and past tense is not as easy without those surrounding clues. One solution is to write out the past inflectional endings (- ed), but there were no examples of this in the collected data. In some cases the caption as a whole helped to determine the tense. An example can be seen from user miamistylemom in (7). Here the eyes emoji ( ) is used for a simple form of the verb to look. The surrounding word units do not signal tense, but the form of the following verb phrase can help with identification. (7) When I in my closet and I feel like I have nothing to wear I just put on a bodysuit and high waist jeans! The following verb phrase is in the present tense. For the sake of cohesion, both verbs will likely have the same present tense. Cohesive ties like these were not always present to help with determining tense though, making the simple aspect problematic. Something else to consider is that not all Instagram users are so careful with their language and may not write out the whole verb phrase on purpose for language economy. In the corpus there was a user who made two similar substitutions, both of which omitted words with crucial information needed to determine person and tense of the substitution. In (8a) below the -ing ending is the only clue present to help identify the form of the whole verb phrase. The caption in (8b) has even less information presented, making the aspect, mood, tense, and person are all unclear. Despite the missing information, the reader is able to generally understand the sense of excitement the writer has about upcoming events which may be satisfactory for the writer. (8) (a) ing into the weekend because it’s finally feeling like Spring! (b) into 2018! Happy New Year!! Verbs were used significantly less than nouns, which was expected because of the complications in illustrating verbs and the more complex grammar system of English verbs. There were no auxiliary verbs substituted, only lexical ones. This may be because the emojis draw attention and graphically highlight certain parts of text, which would not suit auxiliary verbs since are only meant to convey backgrounded grammatical information. Because of how English verb phrases are constructed, substitutions in the
41 progressive and perfective aspect had auxiliary verbs that would identify the person, aspect, and tense of the verb phrase when the emoji substitute did not. When the auxiliary verbs were not present, the grammatical information was not certain and only the general idea of the message could be understood by the reader. The simple aspect also presented a problem since they do not use these auxiliary verbs making the difference between present and past tense difficult to identify. It is possible that these complications discouraged substitution and hindered the number of verb substitutions overall.
6.1.3 Adjectives
Shifting our focus over to adjectives, this part of speech placed third in frequency with 84 instances, making up 9% of all replacements. One of the reasons why they were used so infrequently could simply be because adjectives are not required to make a full sentence, unlike nouns and verbs. Another possibility is that they often express abstract concepts or modify a quality of a noun, which would be problematic for small images like emojis to convey. Typically they “denote properties – most certainly in the domains of size, shape, color, worth, and age” (Huddleston and Pullum 2002, 527). Even simple descriptive examples like old or sweet would be very difficult to visualize or show as an emoji. Looking at the collected Instagram captions, some of the most frequently used emojis stood in for abstract ideas. More specifically, they were words that involved a personal evaluation of something. It did not seem like they were replacing a word in particular but instead captured a concept that one of several words could express. This can be seen with the most frequent emoji to replace adjectives, the 100 points emoji ( ) with 26 instances. Looking through all these recorded substitutions with this emoji, it did not seem to replace a specific word but generally intended to stand in for a positive descriptor, like the words great, perfect, or fantastic. Some examples can be seen below. (9) (a) Cakes with flowers are my favorite kinds. And beaded chandeliers look in the background. (b) I've never met a macaroon topped cake that I didn't like! And this one is 42
(c) Hair flippin good Loved @buddyvs @vegas Everything from the chicken parm to the carbonara was The caption in (9a) above describes a cake and room decorations, (9b) also describes a cake, and (9c) describes 2 other dishes of food. All of these substitutions are preceded by a sentence praising another item. This matches and reinforces the positive sentiment that the emoji gives in the second sentence. The second most frequent emoji was the ok hand emoji ( ) with 101 adjective substitutions. Interestingly the hand gesture, a nonverbal element of face-to-face communication, was used in place of a word. Similar to the 100 points emoji, it was also used as a positive descriptor. Two examples can be seen below, discussing food. (10) (a) Beets, turmeric, onion skins and my personal favorite, red cabbage. It definitely doesn’t smell the greatest when it’s boiling away on your stove but that robin egg blue hue is just . (b) Enjoying some super cozy Egg Drop Noodle Soup with mushrooms, spinach and @peteandgerrys organic eggs today during this ongoing flu season. It only took 10 minutes to make which is ! The third most used emoji, the fire emoji ( ), was sometimes used similarly as a positive descriptor. This emoji was not always used in place of the word fire but occasionally as a positive adjective, more like how people use hot or on fire to compliment a person. (11) (a) M I A M I I’m coming to you #TheMasterClass Miami on September 29th ! link in my bio. It’s gonna be @themasterclass (b) ps: my dress for tonight is The large number of emojis used to describe something positively can possibly be explained. As was mentioned in the theory section of this thesis, emojis are often used to make interaction more fun and positive. Using emojis might be a way to boost positive sentiment even further than words alone could.
1 Total includes all skin tone variations 43
Aside from adjectives expressing a positive sentiment, there were a handful of substitutions describing a quality of the following noun. In one caption the orange and purple heart emojis ( and ) were used when asking which of the pictured outfits looked better, seen in Figure 20. The rainbow emoji ( ) was used similarly to describe how colorful an item was in a few captions, like in Figure 21. In terms of grammatical form, English adjectives are relatively simple. The only inflectional endings that could complicate substitution concern gradation (-er and -est) and these forms were not substituted in the corpus. Adjectives do have some derivational endings which could be problematic for emoji substitution like -y, -al, -ed, and -ful. Adjectives with other derivational endings, like -acious and -able, are also possible but may be too conceptually complex for an emoji to express. There were 3 examples that added in the suffix after the emoji, as was seen with the previously discussed parts of speech. (12) (a) Everything’s just y. (b) dress is @apieceapart from 3 years ago but I put a bunch of similar ones in my instashop if you’re looking for a -y dress! In (12a), the placement of the emoji and -y derivational ending make it clear that the emoji substitutes the adjective “peachy” and there is no need to see the image for context. User graceatwood expresses the -y ending the same way in (12b), albeit with a hyphen. The third instance of a derivational ending being spelled out is seen in Figure 22, although this substitution is within a hashtag and uses nontraditional spelling. The caption shows a dog named Tuna with an unsatisfied expression and wrapped in a towel. When the whole caption and the image of the unhappy dog are considered, the reader can deduce that the crab emoji ( ) and -ie ending combination stands in for the word crabbie, meant to be understood as crabby. (13) Anywhere with weather and architecture is a win in my book There were a few cases where the derivational ending was dropped after substitution. User alyssainthecity substituted the same word as graceatwood in (12b), sunny, but did not signal the derivational ending, seen
44 above in (13). Spelling out the suffix explicitly does not seem to be needed though. The placement of the emoji before the noun weather signals that it is an adjective and the reader can fill in the necessary -y ending easily. Another omission can be seen in Figure 23, where the eyes emoji ( ) makes up part of the phrase blue eyed beauties. The -ed ending of eyed is not explicitly given but can be assumed by the reader. This is because of the placement of the word in the sentence clearly signals that the emoji is used as an adjective. The image content of the post showing two women with striking blue eyes may also give the reader an additional hint that the original wordform was eyed. Interestingly, there were a few examples of adjective substitution with multiplied emojis, like in (14). Adjectives do not express number, so it is not certain why they are multiplied. Perhaps it is to emphasize the adjective, or make it more intense in emotion, like the effect bolding or italicizing would have. The replaced adjective in (14c) also carries this emphasis but is executed with several different emojis expressing positive sentiment instead of repeating the same emoji. (14) (a) London food tour with @tomsbigeats and these donuts @doughnuttime_uk were (b) I’ve got these beach/picnic-ready niçoise sandwiches up on the blog this week that are so (c) It’s pretty here and I have a passion fruit marg so everything is
Before the corpus was compiled it was predicted that adjectives would appear less often than nouns. Adjectives often describe specific features of nouns that would be difficult for emojis to illustrate. There was indeed a small handful of substitutions which modified a specific quality of a noun, like color. Looking at the other examples of substituted adjectives, a large portion of them were used as positive descriptors of a noun, which was unexpected. Emojis are regarded as fun and positive, so they might have been used to boost the positive sentiment of a sentence further than a word could. In terms of grammar, it was predicted that comparative suffixes could be problematic. There were no substitutions in the corpus with these endings which may mean they were too difficult for substitution. Although there was a small handful of instances with
45 derivational endings. In these few cases the endings were either spelled-out or could be assumed through context easily, so the derivational endings did not pose a big challenge in understanding. It is more likely that adjectives were substituted less because of the difficulty in illustrating them and not a complication with grammar.
6.1.4 Adverbs
This part of speech did not trail too far behind adjectives. Adverbs were replaced 61 times in total (7% overall). The substitution of adverbs was unexpected since adverbs are meant to specify how an action was performed, like quickly, which is something that could be very difficult to visually. However looking at the data more carefully, the emojis substituting adverbs were predominantly pointing hands and arrow symbols involving direction. Only 9 out of the 61 emojis did not point in a certain direction. The pointing hand and arrow emojis were used frequently for adverbs because of platform specific reasons. Their function was to direct a user to a specific part of the webpage or mobile app. The emojis pointing right were used to show which direction had more picture content. The emojis pointing upwards were also used a few times to direct the reader upwards to a link in the profile’s about section. The emojis pointing down were used similarly, to direct the user downwards to the comment section where they could write a comment on the post. An example of each of these directions can be in seen in Figure 24, Figure 25, Figure 26, and Figure 27.
6.1.5 Other Parts of Speech
Nouns, verbs, adjectives, and adverbs made up 96% of the substitutions collected for the Instagram corpus. The remaining 4% of substitutions largely included numerals, pronouns, and prepositions. There were 22 captions that used emojis for numerals. While this was not expected, looking at the examples more closely we can potentially see some reasons why the substitutions took place. Thirteen of these substitutions used the pointing up emoji ( ) and victory hand emoji ( ) to stand in for the numbers one and two respectively. Nine of these were from the same user, beautyblender, when discussing beauty products. The one hundred point emoji
46
( ) was used several times to stylize the number 100 when speaking about things like the hot weather (in Fahrenheit). Another can be seen in Figure 28 where the one hundred emoji is referring to the number of a color in the accompanying photograph. The same was done with the ten emoji ( ) on two occasions. In these cases the emoji may have been used instead of text to visually emphasize the number, differentiating it from the other words in the Instagram post. The caption below from user sarahhearts celebrates a wedding anniversary and she uses a number emoji to visually emphasize how long she has been married. The number of the anniversary is a key piece of information, especially for a milestone like 10, and the colorful emoji helps to emphasize that. (15) years today!!! There was also one example of a preposition being replaced. The substitution was a plane emoji between two city abbreviations, seen below in (16) The graphic not only successfully communicated the original to preposition, but also communicated that the mode of travel was by plane, which cannot be specified by the preposition to alone. (16) to @hkairlines’ maiden voyage from SFO HKG Another word class that was not expected to be substituted was pronouns, which largely encode grammatical information. Surprisingly three examples of these substitutions were found in the corpus. All three of them used eye emojis ( or ) in place of the first singular pronoun I, using rebus writing (see 4.2 Rebus). Each example was also followed by a heart emoji and a capital letter U. The captions were meant to be read as I love you. Interjections were substituted 6 times. Four of these were the victory hand emoji ( ), standing in for the word peace in the farewell peace out. User girlgonechildinnyc used the wave emoji ( ) in place of a greeting, which is below in (17). It is clear that the emoji substitutes a word like hello or hey and is not used as surrogate for a nonverbal gesture. (17) I hope they say to all my kids socks too Finally, there was also one example that used the cheese emoji ( ) in place of the exclamation “cheese!” when posing for a photo.
47
More difficult to categorize as a part of speech, there was also an instance where an emoji replaced an expression. The expression was fingers crossed and, as can be seen in (18), the expression is meant to convey hopeful feelings. Unsurprisingly, the substitution was made with the crossed fingers emoji ( ), seen below. (18) Heading back east this weekend and this coat still fits While these parts of speech were not expected to be substituted because of their more abstract meaning, users did still find appropriate situations to replace them with emojis successfully.
6.1.6 In Conclusion of Parts of Speech
The original hypothesis predicted that nouns would be the most substituted part of speech, and this was confirmed. Sixty-nine percent of substitutions replaced noun wordforms, which was at least six times more than any other part of speech. Nouns were predicted to have the highest count because pictographs are ideal for visualizing concrete objects and English nouns are also grammatically simple, meaning few inflectional endings would complicate substitution. Other word classes had more challenges with substitution, either being more grammatically complex or more difficult to visualize through emojis, possibly discouraging their substitution. Although grammatical inflections did pose a challenge, upon analysis it was found that users found alternative ways of expressing the information they gave. Some created a hybrid form, using an emoji for the stem and writing out the endings. This made the grammatical information clear and not subject to guessing. In other cases the endings did not need to be expressed explicitly because of grammatical agreement with surrounding wordforms. For example, with verb the auxiliary verbs and pronouns played an important role in determining person, aspect, and tense of the substituted wordform. Something that could be considered further is emoji substitution in other languages. English has a rigid word order making the identification of word classes relatively easy, but not all languages have this. Languages that encode more grammatical meaning through wordforms could also face big challenges. For now with our data dealing with the English language, we can say that nouns are an ideal candidate for emoji replacement. 48
Something noteworthy that could be considered for further research is the way that gender is expressed through emojis compared to words. Since 2016, many emojis have had a male and female version (Unicode Consortium 2016). This means that dual-gender nouns like dancer, which do not specify gender, could have the gender specified through the emoji’s graphics (female dancer by a long red dress and male dancer by a blue suit).
6.2 Emoji Categories
After closely examining the performance of substitutions by part of speech, we can now make some observations about which emojis were used in substitution. To do this, the emojis used in substitution will be organized into category for analysis. The main goal of this part of the analysis is to see which emojis are used most often in substitution and discuss why that might be. Before collecting data, the hypothesis was that the face emojis, although often ranked as the most used emojis on social media, would be used sparingly in substitution. This prediction was made because of trends found in previous research about emojis in computer-mediated communication. The studies showed that the face emojis more accurately depict nonverbal and paralinguistic elements of communication and their influence on surrounding text is more pragmatic. From this the idea was developed that the face emojis are more fitting for adjunctive use than contributing denotative meaning in substitutions. For this analysis the substitutive emojis were sorted into nine categories: food & drink, objects, people & body, emotion, travel & places, animals & nature, symbols, activities, and face. These categories are based off the categories the Unicode Consortium uses to organize emojis. The only difference is that the Consortium’s smileys & emotion category is split into two categories to better gauge the performance of the face emojis separate from the emotion emojis.
6.2.1 Face
First we will take a closer look at the face category, the focus of the hypothesis. Their use as a substitutive emoji was unexpected because of previous studies on emoticons and expressive emojis and their resemblance to nonverbal
49 aspects of communication. Face emojis did appear as substitutions, but not frequently. Emojis from this category were only used 22 times, making up 2% of the emojis used overall. When the examples are looked at more carefully, many of the substitutions using face emojis do not consider the expression of the emoji at all. The most used emoji was the poo emoji ( ), used 11 times. Each time the emoji replaced a word related to fecal matter like “shitty” or “crap”. The cowboy hat face emoji ( ) and face with sunglasses emoji ( ), and skull emoji ( ) are not used for their facial expressions either, but for the words cowboy, sunglasses, and skull respectively. The sweating smile emoji ( ) also did not focus on the expression, standing in for the verb sweat, seen in (19) below. It may be that the user ignored the facial expression entirely, because there are several other sweat drop emojis with negative expressions that would better complement the message’s negative sentiment (like ). (19) I’m so ready for sweater weather! It has been 100 degrees almost every day the past couple weeks and I’m and that’s definitely not the sparkly kind of sweat...like buckets There were a few substitutions that focused solely on the facial expression of the emoji. Two examples are the vomiting emoji ( ) and nauseated face emoji ( ) were used when discussing vomiting. An example from user katiesturino is shown in (20). (20) Not to make you but you can’t find happiness in someone else, you’ve got to love yourself first! User tunameltsmyheart used the kissing face with closed eyes emoji ( ) with a focus on the expression, shown in Figure 29. The emoji is used to describe a positive reaction fans could have when meeting the dog named Tuna shown in the photograph, like making a kissy face in adoration. Another was the heart eyes emoji ( ), used to express a person’s positive opinion about something. Finally, there was also the cold face emoji ( ) used in place of the word cold. It visualizes the concept of cold through the color blue and icicles. In addition,
50 it also complements the user’s negative opinion about the weather through the distressed expression. (21) back to NYC // not ready for the Of all the emojis used to substitute words, only a small handful were used from the face category. Even less of them were used for their facial expression. This was precisely predicted because of the role of facial expressions in language. In terms of frequency, this category and one other category (activities) were used the least frequently. Each category only had 22 emojis used in substitutions, which confirms the hypothesis that the face emojis would be among the least used. Even though analyzing the face emojis is enough to find evidence of why the hypothesis was confirmed, having a closer look at how the other categories performed provides some interesting information as well.
6.2.2 Food & Drink
This category was used the most in substitutions. The food & drink category also had the largest number of unique emojis used, meaning it had the largest variety of emojis used. Twenty-two percent of emojis used were from this category. The high performance of this category may have been because several of the Instagram blogs focused on food and health. The most used emoji from this category was the hot beverage emoji ( ). Looking more closely at the captions with the coffee emoji, it appeared so often because many users would comment on their personal life and routine which involved coffee. The pizza emoji ( ) was a close second, being used 11 times and the peach emoji ( ) was used 11 times as well. What was unique about the peach emoji was that it was more often used to refer to the buttocks body part instead of the fruit, for example by fitness bloggers when specifying which area of the body a workout focused on. The caption writers may have thought this would be a friendlier way to refer to the body part than using words.
6.2.3 Objects
Emojis from the object category were used with the second highest frequency. Fifteen percent of emojis came from this category (141 emojis). This category also had the second greatest number of unique emojis used. The camera emojis 51
(including with and without the flash, ) were used 40 times combined, a comparatively high number. In most cases the camera emojis were used to refer to the photograph of the Instagram post. For example, most of them were used to remark on who took the photograph or how it was taken, like is shown in (22). The high frequency of camera emojis is likely platform specific because Instagram is focused on image content and requires an image for each post. This emoji may not appear as a substitution so often on other platforms which do not have this focus on images. As for why users decided to use the emoji instead of words, it is not clear. It may have been to save space in the message since the photograph credit is usually only a minor piece of information. (22) (a) missing that LA sun lately, so i just booked a ticket wearing @charleskeithofficial bag and sandals on my last getaway there, by @martinatolot_photo (b) Pretty flowers + simple packaging makes my heart patter. What do you think a gift of flowers says? by Thais Ramos Varela (c) Cheers #Friyay by Natasha Mandic The gem emoji ( ) was also used in many substitutions, occurring 19 times. However, looking at the substitutions more carefully, 15 of these came from a single account focused on jewelry, skewing the data. However what is interesting is that using the gem emoji may have helped to point out the beauty of the precious stones. The bright color and shining surfaces of the emoji’s graphic may help exaggerate the beauty of the diamonds she features in her posts. The user may also use it just to make the message shorter, using a small graphic instead of spelling out a word like diamond. One example of a caption using the emoji is below in (23). (23) Guess that size! Hint...it’s big
Aside from the camera emoji and the diamond emoji, none of the other object emojis appeared a significant number of times.
6.2.4 People & Body
This category was slightly complicated by the fact that many of its emojis come in several skin tones. So there is not just one version of emojis like the 52 clapping emoji ( ) but six including the different skin tones ( ). A user can choose whichever tone they like, and often online users will choose their own as a form of self-expression. Because of this there are 2 counts of unique emojis from this category: counting skin tone variations separately (57) and counting all the variations together as a single unique emoji (38). The two most used emojis from this category were the eyes emoji ( ) with 22 instances and the right pointing emoji ( ) with 21 instances, including all skin tones. The examples using the eyes emojis often replaced an imperative verb like see or look when directing the user to content outside of the post, like a link posted in their profile’s about section. As was discussed before with pronouns, the eye emojis ( and ) were also used as the pronoun I on three occasions (see 6.1.5 Other Parts of Speech). It was not just the right pointing emoji that was used frequently, but other directions (up, down, and left) were substituted with pointing emojis often as well. These hand emojis were primarily used as adverbs to direct the user in a certain direction. See 6.1.4 Adverbs on page 46 for more. This also covers the arrow emojis from the symbol category. There were a few cases where the hand emojis were used as a numeral instead of an adverb. Some of the pointing up emojis ( ) stood in for the numeral one, the victory hand emoji ( ) for two, and the hand with fingers splayed emoji ( ) for five. This may have been a form of highlighting, which is discussed more in 5.4.4 Motivation for Emoji Substitution.
6.2.5 Emotion
There were only 20 different emojis from this category used in substitutions, but certain emojis were used so often that the category became the fourth most frequent across all the data. Most notable is the red heart emoji ( ), which was used 57 times. This emoji was also the most used overall. The red heart emoji was used to substitute several word classes but substituted verbs most. In 36 substitutions the emoji replaced the verb love or another similar verb expressing affection. There are also heart emojis of different colors and slight
53 variations (like the sparkly heart emoji ) that exist. If the other heart emojis are included, then in total there were 52 various heart emojis used to replace the verb love. About half of all the verb substitutions were made by heart emojis, further showing how common this substitution was. The frequency of this substitution today may be attributed to a decades- old slogan. A popular example of this heart symbol substitution, I NY, existed long before emojis were standardized. The slogan was created by Milton Glaser to promote tourism and has been in use since 1977. The New York State Library claims that the heart substitution “has been widely imitated since then” (New York State Library 2018). The one hundred emoji ( ) was also used often with 37 appearances. This emoji was the second most used overall as well. Reasons for why this emoji was used so often are discussed in the following section. The rest of the emojis from this category were only used a handful of times or once.
6.2.6 Symbols
A lesser used category was symbols. There were 42 substitutions that used emojis from this category, and about half were arrow emojis used to direct the user to a specific part of the Instagram platform (see 6.1.4 Adverbs on page 46 for more). Nine of the symbol emojis were used in a unique way because the word they were replacing was written in the emoji itself. This includes the UP! button emoji ( ), TOP arrow emoji ( ), the BACK arrow emoji ( ), the SOON arrow emoji ( ), FREE button emoji ( ), and the NEW button emoji ( ). Of course because these emojis have words in them, they no longer have the visual limitations of pictographs. As for why a person would use these emojis over text, see 6.3.1 Motivation for Emoji Substitution on page 56.
6.2.7 In Conclusion of Emoji Categories
In this section of the thesis I have taken a closer look at which emojis were selected for substitution. The hypothesis was that the face emojis would be used marginally and this was confirmed, with only 2% of substitutions using
54 face emojis. The prediction was based on the idea that the expressions in these emojis are better suited to give information as an adjunct to text than as a replacement for text. Examining the examples of face emojis that did substitute text, many of them were not focused on the expression of the emoji at all. Several of them were more focused on something else that was pictured in the emoji, like a hat or sunglasses. This means that facial expressions were even less popular for substitution than the numbers suggested. Analyzing the other emoji categories revealed some interesting trends. Unsurprisingly, the categories which pictured objects and other concrete entities were used more often in substitution. The emotion category, which largely does not feature concrete entities, had a surprisingly strong performance. This may be because the heart emojis and the one hundred emojis help inject a positive mood into the message, but this will be discussed more in a following section. Some of the emojis were used more frequently only because of reasons specific to the Instagram platform. This was especially the case with the camera emojis and the emojis showing direction. It is very possible that messages on other platforms do not use these emojis as often for substitution. This concludes the analysis of data relevant to the thesis hypotheses. However while examining the corpus to validate the hypotheses, I did notice some trends interesting examples of substitution that are worth presenting as well.
6.3 Miscellaneous Observations
While analyzing the corpus to confirm or deny the hypotheses, I did make some additional observations. In particular, I noticed some trends in message composition, possible motivations for users to use substitutive emojis, and some interesting examples of substitution. While they do fall outside of the scope of analysis, they concern the same background theory and provide an interesting perspective on the data that is not shown otherwise.
55
6.3.1 Motivation for Emoji Substitution
So far, analyzing the complied corpus has given us the opportunity to examine how substitutive emojis are used. Something that should also be addressed is why they are used instead of the words they replace. What is important to remember is that adding in emojis makes the message multimodal. That means there are more forms of communication present to deliver a message. When emojis are used as a substitute it means the modes of communication are in competition. Either text or an emoji is used to perform a part of the communication. Text is much more systematic and can express some complex ideas within a word while other ideas are easier to capture and understand using visuals. In 2.2 Why Use Emojis? I reviewed some of the reasons why people use emojis in general, whether they were adjunctive or substitutive. Several sources discussing emojis determined that people like to use them mainly because they are fun. They are little images that add a pop of color to a message that is otherwise black text. While sifting through the examples of substitutive emojis in the corpus for analysis I did find some trends which could possibly help explain why an emoji would be used over text. The most prominent reasons seemed to be as a light form of censorship, to inject a more positive mood, or to highlight something. Using emojis as a form of censorship seems to be out of politeness, more specifically to avoid saying an uncouth word. One of the more frequent emojis to be used in the corpus was the poo emoji ( ). Each time the smiling piece of dung was used it stood in for a word like crap or shitty. The emoji acts like a form of censorship, using a happy image in place of a rude word. In (24a) we can see that a user went so far as to use it twice in a single sentence. (24) (a) no lie, was having a morning in a string of days (notice the same exact outfit i’ve been wearing all week) and then i came here and talked to a bunch of amazing women about a bunch of exciting projects (b) I feel like Users may have had a similar motivation when using the peach emoji ( ) for the buttocks. It may be used as a kinder or friendlier way of mentioning the body part. In the corpus eight out of eleven of the peach emojis
56 were used to refer to this body part. Some may be intended to replace the vulgar word ass, like in (25a). Many of the peach emojis were used in connection with fitness and getting in shape. In these cases it may not have been censorship so much as to encourage a more positive feeling with the colorful emoji and to help motivate users to exercise, for example like the caption in (25b). (25) (a) A throwback to being in NYC this time last year and freezing my off! (b) Did you see my booty burn YouTube video?! LINK IN BIO! You guys have been asking me for some more moves - so here ya go! There was one curious example of substitution for politeness, in this case working off of a euphemism. The emoji was used as an adjective to describe a dress. More specifically there were two melon emojis ( ) used, seen in Figure 31. Since the reasoning cannot be determined from the caption alone, looking at the picture content is necessary. In the photograph the user is pictured wearing a dress that prominently displays her bosom. Seeing how the dress frames her figure, it may be that the melon emojis are referring to her breasts and the user doubled the emoji because women have two breasts. The emojis could be replacing a word like busty. Aside from censorship, many of the substitutions seemed to be used to add positivity to a caption. The sun emoji ( ), which was used 22 times, was mostly used in a positive sentiment, like in the captions below. (26) (a) It’s hard to take walking selfies in the wind but I had a nice day yesterday in the sunshine and thought I’d share some with you! (b) This was the best little weekend because I spent most of it outside and it reminded me how much happier I am when I get a little , even if it’s just for a quick walk or run. (c) Currently bundled up sipping on celery juice hoping my hands don’t freeze off while writing this caption. Take me back to & The food emojis may have similar motivation. There was a number of food blogs in the corpus. Using a food emoji instead of writing out the word may make the message appear lighter and more fun with the dash of color, appealing more to the readers. This would also explain why so many adjective emojis were used as a positive descriptor (see 6.1.3 Adjectives for examples). 57
Lastly, some emojis may have been used to highlight a part of a sentence. Some examples of their highlighting function can be seen in (27). In (27a) the one hundred emoji is used to visually emphasize the accomplishment of having such a large number of posts. The caption in (27b) the NEW emoji may be used to visually highlight the fact that there was a recently released product readers may not know about yet. (27) (a) I realized we've posted over different photos of posts, ideas and DIY balloon projects from over the past 9 years on the blog. (b) Give your lips the love they deserve with @esteelauder’s Lip Care Collection! Before the corpus of substitutive emojis was created, there was only one known motivation for using emojis and that was to inject a positive mood into a message. After examining the corpus more closely, there was also evidence that could further specify motivation. In some examples emojis were chosen over text out of politeness, to avoid using an inappropriate word. In a way this could be seen as an extension of the previously known motivation. Removing the sharp language and replacing it with an emoji could be a seen as a way of injecting a positive mood. Additionally some emojis seemed to be used to visually highlight a specific piece of information. Something else that may be considered is that the emoji substitutions present information in an easier way than text. This could have been the case with the pointing hand and arrow emojis discussed in 6.1.4 Adverbs on page 46. The authors of the captions could have easily written out the direction (up, down, right, left) but instead decided to use an emoji to show the direction visually. Whether these emojis are more clear or efficient than their text counterpart may be the topic of further work, but there is not enough evidence here to draw any firm conclusions.
6.3.2 Message Composition
As was discussed earlier in part 4 Images Substituting Text, there are a few ways that a message with substitutive emojis can be composed. Users can either use the emojis as pictographs, depicting the original concept, or use rebus writing. In part 4 it was determined that rebus writing took too much
58 effort to encode and decode to be practical and it would be more efficient to use the emojis as pictographs. Looking at the captions collected for the corpus, rebus writing was documented only sparingly with users overwhelmingly preferring pictographic writing. Of all 902 substitutions, only 8 used rebus writing. In 3 cases the substitutions were a form of wordplay, seen in (28) below. These captions used an emoji related the general theme of the message to replace a word. In (28a) the rabbit emoji ( ) stands in for the second part of the word somebody and the theme of the caption is Easter. The wordplay in (28b) involves the taco emoji ( ), which is used to approximately sound like “talk a-”. Lastly, the bee emoji ( ) in (28c) replaces be and keeps to the honey theme. Similarly, user Emrata made a pun as well about her own username in (28d). (28) (a) Some is excited! You guys Easter is next weekend. SO FUN! (b) bout an amazing weekend @Austinfoodwine I ate so many tacos, did a cooking demo with @jasongiagrande @heb and ate all the BBQ with @blondeswhoeat (c) Cheesy honey butter biscuits. You would want to all up in these too (d) Em a A few of the rebus substitutions used eye emojis ( or ) to stand in the for the pronoun I, followed by a heart emoji and capital letter U. This combination reads as the phrase I love you. While there were a few examples of rebus writing, it seemed like users preferred using the emojis as pictographs. There were two other approaches mentioned in part 4 related to message composition. When using substitutive emojis, users could choose between using only emojis (transcribing all the text into emojis) or mixed textuality (using text and emojis together). It was reasoned that using only emojis to compose a text is too difficult for readers to interpret since there is no universal way to read an emoji-only passage. They also have a large amount of what Danesi calls “visual noise”, making it difficult for a reader to orient themselves. Although Danesi does also mention that a series of emojis can be
59 used more conceptually to express a general idea without relying on any sort of language system, but with complications. There were no examples of emoji- only captions like this in the corpus, but there was one substitution that exhibited elements of this. This substitution will be discussed more in 6.3.3 Selected Examples of Unique Emoji Substitutions. Mixed textuality on the other hand would frame the substitutions within a language system that readers would be familiar with, making the text easier to digest. Captions compiled for the corpus showed that users much preferred using mixed textuality. There were merely two captions which did not have text alongside the substitutive emojis, seen in (29). The first reads as back soon, one emoji for each word. The second uses two emojis for the compound word bookworm. (29) (a) (b) Emoji-only captions only appeared exceptionally. The two examples that were found kept it simple and limited the substitution to two emojis, making them easier to comprehend. The example in (29a) used emojis with words in the images, making the interpretation straightforward. However overall it is clear that users prefer to use the mixed textuality approach with substitutive emojis instead of relying on emojis alone.
6.3.3 Selected Examples of Unique Emoji Substitutions
While collecting data to verify the hypotheses, there were several substitutions that were interesting either for their creativity or unique formatting that are worth mentioning. In a handful of cases there were substitutions that used several emojis in sequence. These sequences were meant to be read collectively as one word. The user psimadethis has an example like this to discuss an array of plants pictured in her sink, seen in Figure 30. Instead of writing out the word plants she used several emojis of various plants to get the same idea across. Another example used several transportation emojis. These were meant to be read collectively as travel. (30) happily staying put this weekend... no for me!
60
There were also two posts where an extraordinarily high number of emojis was used. The first example is from a photographer asking readers about their favorite method of travel, quoted below in (31). The user lists 6 emojis, like the plane and train emoji, as suggestions for answers. The carousel horse ( ) and rocket ( ) emojis are strange suggestions for transport but are likely only added for humor. (31) What’s your favorite way to travel, , , , , or ? There was one caption by user alyssainthecity with a stunning substitution using 27 emojis. Back in part 4 of this thesis I mention the possibility of a series of emojis standing in for a bigger concept instead of specific language. The caption shown in (32) is the closest example to this in the corpus. The emojis stand in for the verb of the sentence but the series seems to say much more than a single verb could. The series of emojis describes some of the moments the user shared with a friend while traveling. It is an interesting, succinct way to give a general impression of their trip instead of recounting each activity they did through text. (32) no one else i’d rather
with. that’s a wrap on the trip of a lifetime with @blindsaay Finally, there were two substitutions that worked on a different level than all the other substitutions. The first is also from user Emrata, who had the caption in (33a). The eyes emoji substitutes the word looks on two levels. The emoji substitutes the word looks in terms of meaning and the eyes also substitute two of the letters in the word itself. User samhorine also had a multilevel substitution with the moon emoji. (33) (a) Summer l ks (b) M NS over Manhattan
61
7 Conclusion
The aim of this thesis was to fill in a large gap of knowledge regarding emojis and language. While many linguists have taken an interest in emojis and considered some of their functions within language, the available research on the topic was very constrained. All studies concerning emojis only collected data of their adjunctive use to text, not substitutive. Marcel Danesi recognizes that emojis can also be used as a substitute for text and highlights some extraordinary examples. However there is a clear lack of tangible data of substitutive emojis in practice. Because of this, it was not certain what parts of language were substituted, how, or why. The available studies also largely only consider the face emojis, which only make up a small amount of the 1,262 existing emojis. Reviewing literature showed evidence that the face emojis were used more adjunctively while the other emojis were used more often as a substitute for text, but it could not be confirmed. To fill in this knowledge, I consulted similar kinds of visual language and examined their limitations in expression. After comparing those findings with substitutive emojis, I hypothesized that emojis would substitute nouns the most. I also hypothesized that the face emojis would substitute language less than other categories, since literature suggested that they were better suited for adjunctive use. In order to confirm or deny these predictions, I compiled a corpus of substitutive emojis in practice from Instagram. Results showed that nouns were indeed replaced the most and with a large margin, six times more frequently than any other part of speech. Upon analysis it was determined that nouns were likely substituted so often because they are the easiest to visualize and had less grammatical complications compared to other parts of speech. The face emojis were also used only a small number of times for substitution, confirming the second hypothesis. While analyzing these emojis in practice, there were also some observations made about the motivation for using emojis this way, the composition of the messages, as well as a discussion about some unique examples of substitution. With that analysis and discussion, the lack of
62 information about substitutive emojis could be filled at least a small amount, narrowing the gap in knowledge. However there is still much that can be discovered about substitutive emojis that this thesis does not cover. This study of substitutive emojis was only conducted in English, but a study comparing how emojis replace text in another language system could also prove interesting. There could also be observations made about how gender modifiers are used when substituting language. Further, the motivation behind using substitutive emojis could be narrowed down, as well as many other topics. For now with the knowledge we have, at the very least we can say that emojis provide an interesting mode of communication, making interaction just a bit more fun ✨
63
8 Data Tables
User Noun Adjective Adverb Verb Other TOTAL alyssainthecity 23 10 8 1 1 43 asubtlerevelry 27 5 4 4 2 42 beautyblender 12 1 3 8 9 33 belkazan 4 0 1 3 0 8 blaireadiebee 18 2 0 1 0 21 brahmino 6 0 0 1 2 9 cheatdayeats 58 7 0 6 0 71 desiperkins 17 1 1 0 0 19 doctor.mike 1 0 0 1 0 2 dollyandoatmeal 5 2 1 0 0 8 emrata 12 0 2 2 3 19 esteelalonde 17 1 0 0 0 18 gigihadid 11 0 0 0 0 11 girlgonechildinnyc 16 2 0 0 1 19 graceatwood 21 5 0 2 0 28 hannahbronfman 17 1 2 3 0 23 havejewelswilltravel 29 1 5 1 0 36 heymamaco 9 2 2 1 0 14 honestlywtf 12 3 1 3 1 20 ispydiy 3 1 0 1 0 5 jenselter 15 1 0 4 0 20 julieskon 2 0 0 2 0 4 karliekloss 17 3 1 4 4 29 kateleemakeup 8 2 0 1 1 12 katiesturino 38 5 1 6 0 50 kelsibrazell 9 0 0 1 0 10 lira_galore 4 1 0 1 0 6 makeupbymario 12 5 5 4 0 26 miamistylemom 54 0 0 8 4 66 mskateegrace 4 0 0 2 2 8 psimadethis 49 10 2 13 2 76 samhorine 13 0 1 0 0 14 sarahhearts 19 0 0 0 4 23 soyonan 6 0 0 3 0 9 spoonforkbacon 15 4 1 0 0 20 tunameltsmyheart 8 1 6 2 0 17 veenacrownholm 18 8 14 7 3 50 vegas_nay 6 0 0 0 0 6 wckitchen 3 0 0 4 0 7 TOTALS 618 84 61 100 39 902 Table 4: The number of substituted parts of speech by user account
64
Food & Drink Total emojis 210 Unique Emojis 67 (22%) (22%) 3 11 6 3 3 2 2 2 2 4 2 3 2 5 4 6 3 3 3 7 3 9 5 4 2 11 8 5 4 2 2 3 3 3 2 2 2 2 5 15 2 7 7 2 2 Objects Total emojis 149 Unique Emojis 49 (15%) (16%) 9 19 3 5 3 2 4 4 21 16 3 2 6 6 2 2 5 3 2 2 2 People & Body Total emojis 145 Unique Emojis 382 (15%) (12%) 2 2 2 2 2 2 2 2 3 22 2 6 2 2 3 21 4 3 8 2 2 5 2 3 7
Emotion Total emojis 142 Unique Emojis 19 (15%) (7%) 57 3 3 3 6 3 2 4 3 3 4 37 6 3 Travel & Places Total emojis 134 Unique Emojis 38 (14%) (12%) 22 3 4 17 3 10 3 11 2 9 5 13 2 2 2 2 2 2 Animals & Nature Total emojis 102 Unique Emojis 47 (11%) (15%) 5 7 2 2 4 5 5 5 6 3 2 10 2 2 2 3 3 3 2 2 Symbols Total emojis 42 Unique Emojis 20 (4%) (7%) 3 2 2 3 2 13 2 ✅ ❓ 2 2 Continued on next page
2 This number excludes skin tone variations 65
Face Total emojis 22 Unique Emojis 11 (2%) (4%) 11 2
Activities Total emojis 22 Unique Emojis 15 (2%) (5%) 3 2 2 ✨ 2 2 2 ALL EMOJIS Total emojis Total unique emojis 968 304
Table 5: This table lists all the emojis used in substitution in the compiled corpus by emoji category. Emojis that were used more than once have a number next to them displaying how many times they appeared in the corpus.
66
9 Examples
Figure 15: A caption showing how a plural word can be replaced by several repeated emojis in order to visually communicate number
Figure 16: An emoji replacing a singular noun, signaled by the preceding determiner “a”
67
Figure 17: An emoji replacing a singular noun, the number implied through common knowledge
Figure 18: An example of when the plurality could not be determined through the caption alone and relied on the image content for determining number
68
Figure 19: The only example of an emoji standing in for a noun stem with a derivational morpheme spelled out
Figure 20: Using emojis to replace descriptive adjectives, something that showed up rarely in data
69
Figure 21: The rainbow emoji as an adjective
Figure 22: An example of an emoji and spelled-out derivational ending used in combination for an adjective
70
Figure 23: An emoji substitution where the derivational ending can be assumed, even though it is lost in substitution, thanks to the placement of the emoji and image context
Figure 24: A directional emoji replacing an adverb directing the user in using the platform
71
Figure 25: A directional emoji replacing an adverb directing the user in using the platform
Figure 26: A directional emoji replacing an adverb directing the user in using the platform
72
Figure 27: A directional emoji replacing an adverb directing the user in using the platform
Figure 28: One of the few examples where an emoji replaced a numeral
73
Figure 29: An emoji featuring a facial expression being used as a verb
Figure 30: A series of emojis used to stand in for a single concept, in this case plant life emojis for “plants”
74
Figure 31: a doubled emoji standing in for an adjective, through context may be visually representing a women's breasts
75
10 References
Apple. 2017. "Differential Privacy." Apple. https://images.apple.com/privacy/docs/Differential_Privacy_Overvie w.pdf. Benenson, Fred. 2009. Emoji Dick. September 19. Accessed November 18, 2019. https://www.kickstarter.com/projects/fred/emoji-dick. Biber, Douglas, Susan Conrad, and Geoffrey Leech. 2002. Logman Student Grammar of Spoken and Written English. Harlow: Pearson Education Limited. Blagdon, Jenn. 2013. "How Emoji Conquered the World. The Story of the Smiley Face from the Man Who Invented it." The Verge, March 4. http://www.theverge.com/2013/3/4/3966140/how-emoji-conquered- the world. Busche, Laura. 2016. Meet the Graphic Designers Behind the Emojis We Love. May 2. Accessed October 23, 2018. https://creativemarket.com/blog/meet-the-graphic-designers-behind- the-emojis-we-love. Cherelus, Gina. 2016. "Move over Picasso: first-ever emojis to hang in N.Y. Museum of Modern Art." Reuters. October 26. Accessed June 20, 2019. https://www.reuters.com/article/us-usa-new-york-emojis- idUSKCN12Q2SQ. Crystal, David. 2006. Language and the Internet. 2nd. Cambridge: Cambridge University Press. Danesi, Marcel. 2017. The Semiotics of Emoji. London: Bloomsbury Academic. Dresner, Eli, and Susan Herring. 2010. "Functions of the nonverbal in CMC: Emoticons and illocutionary force." Communication theory 20 (3): 249-268. Emogi Research Team. 2015. 2015 Emogi Report. Emogi. Accessed October 12, 2018. http://cdn.emogi.com/docs/reports/2015_emoji_report.pdf. Emojisaurus. n.d. Early to bed and early to rise... Accessed July 18, 2019. https://emojisaurus.com/phrases/22-early-to-bed-and. —. n.d. I'm busy getting ready. Accessed July 18, 2019. https://emojisaurus.com/phrases/6102-i-m-busy-getting. —. n.d. Mount Kilimanjaro. Accessed July 18, 2019. https://emojisaurus.com/phrases/6902-mount-kilimanjaro. Evans, Vyvyan. 2017. The Emoji Code: The Linguistics Behind Smiley Faces and Scaredy Cats. New York: Picador. Garrison, Anthony, Dirk Remley, Patrick Thomas, and Emily Wierszewski. 2011. "Conventional Faces: Emoticons in Instant Messaging Discourse." Computers and Composition 28 (2): 112-125. Gruber, Helmut. 2013. "Mailing List Communication." In Pragmatics of Computer-Mediated Communication, by Susan Herring, Dieter Stein and Tuija Virtanen, 55-82. Berlin: De Gruyter Mouton. Hsieh, Sara H., and Timmy H. Tseng. 2017. "Playfulness in mobile instant messaging: Examining the influence of emoticons and text messaging on social interaction." Computers in Human Behavior (69): 405-414.
76
Huddleston, Rodney, and Geoffrey K. Pullum. 2002. The Cambridge Grammar of the English Language. Cambridge: Cambridge University Press. 2010. Instagram Info Center. October 6. Accessed April 10, 2019. https://instagram-press.com/blog/2010/10/06/instagram-launches-2/. Kavanagh, Barry. 2016. "Emoticons as a medium for channeling politeness within American and Japanese online blogging communities." Language & Communication 48: 53-65. Kress, Gunther. 2012. "Multimodal Discourse Analysis." In The Routledge Handbook of Discourse Analysis, edited by James Paul Gee and Michael Handford, 35-50. New York: Routledge. Melville, Herman. 2010. Emoji Dick. Edited by Fred Beneson. Accessed November 12, 2018. http://czyborra.com/unicode/emojidick.pdf. Moschini, Ilaria. 2016. "The “Face with Tears of Joy” Emoji. A Socio- Semiotic and Multimodal Insight into a Japan-America Mash-Up." Hermes – Journal of Language and Communication in Business (55): 11-25. New York State Library. 2018. New York State Education Department. September 17. Accessed April 25, 2019. http://www.nysl.nysed.gov/emblems/iluvny.htm. Novak, Petra Kralj, Jasmina Smailović, Borut Sluban, and Igor Mozetič. 2015. "Sentiment of Emojis." PloS one 10 (12): e0144296. Ottenheimer, Harriet Joseph. 2008. The Anthropology of Language: An Introduction to Linguistic Anthropology. Second. Cengage Learning. Pardes, Arielle. 2018. The WIRED Guide to Emoji. February 1. Accessed July 12, 2018. https://www.wired.com/story/guide-emoji/. Schenker, Marc. 2016. The Surprising History of Emojis . October 11. Accessed July 12, 2018. https://www.webdesignerdepot.com/2016/10/the-surprising-history- of-emojis/. Skovholt, Karianne, Anette Grønning, and Anne Kankaanranta. 2014. "The communicative functions of emoticons in workplace e-mails." Journal of Computer-Mediated Communication 780-797. Smith, Aaron, and Monica Anserson. 2018. Social Media Use in 2018. March 1. Accessed April 19, 2019. https://www.pewinternet.org/2018/03/01/social-media-use-in-2018/. Stillman, Alan. 2013. "Method for communicating using pictograms and optionally text." U.S. Patent 8,616,887. Washington, DC: U.S. Patent and Trademark Office, December 31. Sugiyama, Satomi. 2015. "Kawaii meiru and Maroyaka neko: Mobile emoji for relationship maintenance and aesthetic expressions among Japanese teens." First Monday 20 (10). Tettegah, Sharon Y, and Martin Gartmeier. 2016. Emotions, Technology, Design, and Learning. London: Academic Press. Unicode Consortium. 2019. Emoji Counts, v12.1. October 21. Accessed November 30, 2019. https://unicode.org/emoji/charts-12.1/emoji- counts.html. —. 2016. "Expanding Emoji Professions: Reducing Gender Inequality." Unicode Consortium. Accessed November 27, 2019. http://unicode.org/L2/L2016/16160-emoji-professions.pdf.
77
—. 2018. Frequently Asked Questions. March 9. www.unicode.org/faq/emoji_dingbats.html. Wolf, Alecia. 2000. "Emotional expression online: Gender differences in emoticon use." CyberPsychology & Behavior 3 (5): 827-833. Yuasa, Masahide, Keiichi Saito, and Naoki Mukawa. 2011. "Brain Activity When Reading Sentences and Emoticons: An fMRI Study of Verbal and Nonverbal Communication." Electronics and Communications in Japan 94 (5): 17-24. Yule, George. 2010. The Study of Language. Fourth. Cambridge: Cambridge University Press.
78
11 Resumé
Počítačem zprostředkovaná komunikace umožnila v průběhu posledních desítek let mnohem snadnější a rychlejší spojení a sdružila uživatele okolo celého světa. Na začátku to byl jen text, ale časem se vyvinuly další komunikační způsoby, například obraz a zvuk. Dalším vývojem přibyly nyní velmi používané emoji. Emoji jsou malé barevné obrázky, který mohou být přidány do textových zpráv, jako například smajlík ( ). Dosáhly velké popularity, ale málo se o nich ví z hlediska lingvistiky. V publikaci The Semiotics of Emoji se Marcel Danesi tento nedostatek snažil doplnit. Upřesňuje zde dva způsoby návaznosti emoji na text. První je doplňující (anglicky adjunctive) a další nahrazuje text (anglicky substitutive). Adjunctive emoji pouze doplňují to, co je už vyjádřeno textem. Na toto téma je k dispozici několik studií. Dospělo se k názoru, že tyto emoji svým způsobem nahrazují paralingvistické vlastnosti jazyka a částečně napodobují neverbální komunikaci. Substitutive emoji naopak text nahrazují. Existuje mnoho studií o adjunctive emoji, ale žádná není zaměřena na substitutive emoji. Je všeobecně známo, že emoji mohou nahradit slova, ale toto není hlouběji prozkoumáno. Cílem této diplomové práce je shromáždit data, doplnit chybějící informace o substitutive emoji a analyzovat, jak se používají v praxi. Na základě průzkumu podobných fenoménů se dá předpokládat, který slovní druh je snadno nahraditelný. První hypotéza je, že emoji nejčastěji nahradí podstatná jména, protože jdou jednoduše graficky ztvárnit a mají gramaticky jednoduchý tvar. Druhá hypotéza se zabývá volbou emoji při nahrazování slova. Výsledky studií naznačují, že emoji výrazů tváře jsou používány zcela jinak než ostatní emoji. Konkrétně studie naznačují, že tyto emoji výrazů spíše text doplňují, zatímco ostatní text nahrazují. Na základě těchto tendencí se předpokládá, že tyto výrazové emoji nahrazují text nejméně. Druhá, praktická část diplomové práce je zaměřena na korpus substitutive emoji a jejich analýzou v praxi. Korpus tvoří více než 900 příkladů substitutive emoji vyhledaných na Instagramu. Z nich je provedena analýza, která potvrzuje obě hypotézy. Z hlediska slovního druhu jsou nejvíce
79 nahrazována podstatná jména, což potvrzuje první hypotézu. Rovněž druhá hypotéza o volbě emoji je potvrzena, protože emoji výrazů tváře nahradily text jen ve dvou procentech příkladů z korpusu. Z toho všeho vyplývá, že emoji doplňuje a případně nahrazuje text, čímž umožní zajímavý čtenářský zážitek.
80