Proceedings of the 50th Hawaii International Conference on System Sciences | 2017

“Nice Picture Comment!” Graphicons in Comment Threads

Susan C. Herring Ashley Dainas Indiana University Bloomington Indiana University Bloomington [email protected] [email protected]

Abstract what extent graphicons are used to convey meaning in public Facebook comment threads. Facebook has increasingly incorporated graphical Facebook lends itself well to graphicon analysis, in means of communication such as emoticons, emoji, that its interface allows users to employ all six stickers, GIFs, images, and videos (‘graphicons’) into graphicon types in their private messages and all comment threads. Adapting methods of computer‐ except GIFs in comments on posts on Facebook mediated discourse analysis, we analyze the frequency Profiles, Pages, and Groups. This sets Facebook apart and pragmatic functions of each graphicon type in from other multimodal communication platforms such threads sampled from public graphicon-focused as Tumblr and , where some of the graphicon Facebook groups. Six main functions emerged from the types are either unavailable, or not commonly used. data: mention, reaction, tone modification, riffing, In what follows we first situate our investigation in action, and narrative sequence. Reaction was most relation to previous research on the use of graphical common, and emoji expressed the widest array of elements in digital conversation and identify the unique functions. We propose structural, social, and technical contribution of this study. We then describe our dataset explanations for variation in graphicon use, and of comment threads collected from public graphicon- suggest improvements for the design of conversational focused Facebook groups, along with our method- graphical elements in systems. ology, which employs computer-mediated discourse analysis [8] to identify the pragmatic functions of the graphicons in context. Six main functions are identified 1. Introduction and illustrated: mention, reaction, tone modification, riffing, action, and narrative sequence. Reaction was Recently the Oxford Dictionaries chose the Crying the most common function, and emoji were most Tears of Joy emoji as the 2015 word of the year, prevalent and expressed the widest array of functions, explaining that “emoji have come to embody a core whereas the other graphicon types tended to specialize aspect of living in a digital world that is visually for certain functions. We propose explanations for driven, emotionally expressive, and obsessively these findings in terms of the graphicons’ structural immediate.”1 Considering this graphical symbol as a properties, ease of use, and history; we also consider ‘word’ is in line with recent popular speculation that social factors associated with their use. In concluding, emoji (in Japanese, ‘picture character’) are evolving we identify challenges that arise in graphicon analysis into a language of their own [24] – if not a complete and suggest changes that could be made to improve the grammatical system, at least a set of signs that can be design of graphical social media platforms. used to convey propositions in conversational exchanges. 2. Background In this paper, we analyze the conversational uses of emoji and five other types of graphical devices found Emoticons formed from ASCII characters have on contemporary social media platforms: emoticons, been used in computer-mediated communication stickers, GIFs, images, and videos. As a shorthand for (CMC) since 1979 [4]. The general findings of this set of devices, we introduce the term ‘graphicons,’ emoticon studies indicate that they are multifunctional a blend of ‘graphical’ and ‘icons’ (cf. Greek grafikon [27], having at least four uses in CMC that potentially ‘graphics’). In particular, we investigate how and to overlap: (1) expression of emotion [4], (2) nonverbal signaling [1, 3, 14], (3) tone management or indication of illocutionary force [4, 12, 14], and (4) as 1 http://blog.oxforddictionaries.com/press-releases/announcing-the- punctuation or structural markers [1, 12, 20]. oxford-dictionaries-word-of-the-year-2015/, retrieved June 14, 2016.

URI: http://hdl.handle.net/10125/41419 ISBN: 978-0-9981331-0-2 CC-BY-NC-ND 2185 Initial research on emoji suggests that they fulfill Little research has investigated how videos are similar roles as emoticons, although emoji are more included in ongoing conversational contexts, although visually complex and may be expected to function [9] investigated how young girls use video messages to somewhat differently in CMC as a consequence. For chat in VideoPal, an asynchronous communication example, [7] found that emoji had a greater effect than system designed around the exchange of videos, and emoticons on reader perceptions of a writer’s [19] analyzed a religious debate that occurred through commitment and personal mood. At the same time, the dyadic exchange of videos on YouTube. there are important limitations to emoji use, due in part GIFs are also understudied, although what research to the fact that different viewing platforms render has been done is suggestive. The preliminary analysis emoji differently. [16] found that people often of Tumblr posts carried out by [2] found that animated disagreed on the sentiment and meaning of the same GIFs typically expressed reactions to previous visual representation of an emoji, and these propositions, and they expressed more emotion, more disagreements only increased when the “same” emoji intensely, and were more positive in valence than text. were compared across platforms. Similarly, [18] characterized exchanges involving Research on photographic images typically does reaction GIFs and images in a Sherlock fan group on not focus on their conversational functions [e.g., 25, Tumblr as conversational interaction. 26]. However, there are some notable exceptions. [28] One of the very few papers to consider stickers [13] examined the use of personal photographs in online suggested that stickers, together with photographs, chat. [15] analyzed a community image blog and found videos, and emoji, function to improve the six conversation styles, including image quote and interpretability of messages and help users express text-in-picture. [6] studied Radar, a mobile application complex emotions. At the same time, the author noted that allows users to post personal photographs, (p. 3) that stickers can lend instant messages “an air of including ‘selfies,’ to a private list of invited friends. equivocation, allowing the conversation to be shaped Users of Radar sometimes exploited the chronological by the different parties as it went along.” This nature of the app to post sequences of images meant to observation recalls the findings of [16] regarding the tell a story. [5] found that images on a messaging potential ambiguity of emoji. application were woven into the thread of conversation There is thus considerable evidence that each meaningfully, rather than simply being mentioned. graphicon type can function pragmatically in CMC, Another study [10] explored how selfies are although little research has taken a conversational or perceived and used in the U.S., U.K., and China discourse approach to graphicon use. Moreover, most through surveys and interviews. The respondents previous studies have analyzed graphicon types reported that selfies often elicited comments and individually, rather than comparing across types. In encouraged conversational partners to send their own this study, we employ a discourse-pragmatic approach selfies in reply. U.S. respondents also indicated a high and systematically compare multiple graphicon types enjoyment of conversational partners ‘playing on’ in order to understand how they function in relation to shared images, especially on Snapchat. one another in conversational threads. An is a “particular idea presented as Specifically, we address two research questions: a written text, image, language ‘move,’ or some other RQ1: How often are different graphicons used in FB unit of cultural ‘stuff’” that is taken up and spreads comment threads in groups devoted to rapidly [11, p. 202]. Considerable research has been graphical content? done on how memes form and spread [e.g., 22, 23], but there has been less research on how such memes are RQ2: How do different graphicons function in FB used in conversational exchanges. An exception is comment threads in groups devoted to [29], which explores CAHOOTS, a chat system that graphical content? continuously analyzes participants’ chat and suggests relevant humorous images and internet memes. In 3. Methodology comparison to random image insertion and plain text chat, users preferred using CAHOOTS. They felt that 3.1. Data the way the system allowed human and computer to riff off one another enabled them to express their The use of multiple graphicon types in Facebook unique sense of humor. Another study analyzed how comment threads is a relatively new and as yet not internet memes were used in the Occupy Wall Street widespread phenomenon. To locate groups with a rich movement [17]. It found that image memes facilitated concentration of graphicons to analyze for the purpose conversation from divergent perspectives and increased of this study, we searched in Facebook for the the accessibility of the discourse. keywords (ASCII) Emoticons, Emoji/Smileys,

2186 Stickers, GIFs, and Images/Memes.2 No groups on the pragmatic functions of each type of graphicon for each topic of ASCII emoticons were found, but we sampled thread and group, taking into account the discourse two public groups from each of the remaining four context surrounding each instance of graphicon use. categories, for a total of eight groups. See Table 1.3 All 975 instances of graphicon use in the dataset were analyzed. (Facebook’s ‘reaction’ emoji, which became Table 1. Public Facebook groups sampled globally available in late February 2016, in the middle of our data collection period, were not analyzed, GIFs EmojiXpress Anime GIFs Smiley because at the time they could not be used in comments, but rather only to react to posts.) Grumpy Cat Memes Stickers We adopted a grounded theory approach to allow Nihilist Memes Stickers FB the function categories to emerge from the data, and succesively refined the operationalization of each From each of these groups, we sampled three recent category through iterative coding. The interpretation of threads between January and May 2016 based on the some graphicon usage was subjective, however, and number of comments and variety of graphicons they coding was made more challenging by the fact that contained. The posts that served as prompts for the each group used graphicons somewhat differently. In threads were dated between January 29, 2014 and May order to identify all the functions attested in the data 15, 2016. The 24 threads included 2,888 comments and and reach saturation in our coding categories, the two 975 graphicons. Of these, 527 graphicons were used by authors ended up coding all of the data jointly, with females, 377 by males, 7 by individuals identifying disagreements resolved through discussion until with a gender neutral pronoun, and 64 by non-personal consensus on the most likely interpretation of each accounts.4 Gender was determined by following the graphicon in context was reached. link from a commenter’s userID to their Facebook profile, where each user is referred to as either 4. Findings ‘she/her,’ ‘he/him,’ ‘they/them,’ or as a community (non-personal). 4.1. Pragmatic Functions of Graphicons Perhaps due to the global appeal of images, the The main pragmatic functions that emerged through threads included many comments in languages other our analysis are: mention (vs. use), reaction, riff, tone than English. We were able to translate most of the modification, action, and narrative sequence. In non-English comments,5 especially since similar addition, there were some ambiguous uses, and a few content was produced across languages. To check our other uses. Each of these functions is operationalized translations, and for languages we did not know, we and illustrated in this section. used Google Translate, the ‘translate this’ feature of Following the classic distinction in analytical Facebook, and/or we consulted native speakers. philosophy of mention versus use [21], we first

identified simple mentions of graphicons. These refer 3.2. Analysis Methods to the graphicon itself, in contrast to communicative

An assumption of this study is that each graphicon uses of a graphicon. Examples 1-3 are graphicon occurrence potentially expresses meaning in occurrences that were coded as mentions. conversational interaction, where ‘conversation’ is 1) [Grumpy ; Sourpuss thread; emoji and image] operationalized as message exchange in asynchronous comment threads. Computer‐mediated discourse analysis, or “language-focused content analysis” [8, p. 4], was employed to analyze the frequency and

2 We did not plan to analyze videos at first, so we did not search for video groups. However, since some videos were found in the comment threads, we decided to include them in our subsequent analyses. 3 The quote in the title of this paper is from a comment posted to a group we did not select, Rough Roman Memes. 4 No claims are made regarding the generalizability of the frequency distributions in our data to Facebook groups as a whole, since our sampling procedure employed judgment criteria rather than systematic sampling. (Cat emoji and photo are both ‘mentions’ of Grumpy.) 5 The first author had studied 12 languages as a doctoral student in linguistics.

2187 2) [Cat GIFs; Kisses thread;6 emoji] “Pancakes are too sweet for the bitter pain consuming my heart,” triggered the riffs in examples 7-9.

(Emoji duplicate kissing motion implied by ‘muah.’) 7) [Nihilist Memes; Pancakes thread; video]

3) [EmojiXpress, Purple theme thread; sticker] I need this one. Srsly.

(‘This one’ refers to the flag face sticker.)

(extends idea of pancakes in prompt to waffles) All of the non-mention graphicon functions fall under the broad rubric of ‘uses.’ 8) [Nihilist Memes; Pancakes thread; sticker] The first of these, reaction, was operationalized as a graphicon use that depicts an emotional response to content that was posted earlier in the thread, typically in the initial prompt. Examples 4-6 illustrate reactions. The emotions expressed are great happiness, gloom, and amazement, respectively.

4) [Cat Memes; Booty cat thread; emoticons] (reverses idea of [not] eating pancakes in prompt to =)))))))))))))) :))))))))))))))))))))) [cat] eating pancakes)

5) [Anime GIFs; Goat thread; sticker] 9) [Nihilist Memes; Pancakes thread; image]

6) [Anime GIFs; Goat thread; image]

Text: ‘Pudding can’t fill the emptiness inside me. But it’ll help.’ (parodies idea of pancakes and emptiness in prompt as pudding and [partial] emptiness)

In contrast to stand-alone graphicons, tone modification is graphicon usage that directly modifies the text it accompanies. The graphicon functions as a nonverbal, paraverbal, or paralinguistic cue as to how Another type of response is riffing, a humorous the text should be interpreted. This includes the use of elaboration on, play on, or parody of a previous graphicons to clarify intent and hedge the illocutionary graphicon or text comment. A prompt in the Nihilist force of an utterance [cf. 4]. Consider examples 10-12: Memes group, an anime-style image of a man with a bitter expression on his face and the superimposed text 10) [StickersFB; Opi thread; emoji]

6 Names of threads in examples were assigned by the authors. (in a discouraged manner)

2188 11) [Smiley; Lantern thread; emoji] meanings. The graphicons in User5’s comment in example 18 could be interpreted in several ways.

(French) ‘It’s too beautiful’ 18) [Cat GIFs; Kisses thread; emoji] (with intense emotion, fighting back tears)

12) [EmojiXpress; Blue theme thread; emoticon] The middle finger is a welcome guest xD (delightedly)

An action is a graphicon used to portray a (typically) physical action. An action can sometimes substitute for the predicate in a text comment, as in the heart to mean ‘love’ in example 13. It can add a nuance of meaning, as do the praying hands in 14, or stand alone as a proposition, as in 15 (offering a rose). In response to a prompt of two kittens “kissing”, User5 posted in Spanish “But what cute kitties!” followed by 13) [EmojiXpress; Blue theme thread; emoji] a ‘see no evil’ monkey and a bear. What is the function of these two emoji: Are they riffs (other cute animals)? Or is the monkey a reaction, and the bear intended as a cat (mention)? The context of the thread does not 14) [Smiley; Mug thread; emoji] provide sufficient cues to disambiguate. Finally, a code of other was assigned for graphicon functions that did not fall into one of the above categories. An example is the use of repeated arrows7

to point to the user’s “favorite” emoji mentioned in ex. (in response to the prompt: “Who makes your life 19. (The two smirking emoji on the right express tone.) beautiful?”

19) [EmojiXpress; Emoji stickers thread; arrows] 15) [Anime GIFs; Bad boy thread; sticker]

4.2. Frequency Distribution of Graphicons

Table 2 displays the frequency distribution of the pragmatic functions of graphicons found in the 24

comment threads. Reactions were most frequent, A narrative sequence is a series of consecutive followed by tone modification and then mentions. Riffs graphicons that tells a story of sorts. Two examples and actions were less common; sequences and other (along with approximate verbal glosses) are given in uses were infrequent; and ambiguous cases were rare. 16 and 17. Table 2. Frequency of graphicon functions 16) [StickersFB; Rilakkuma thread; emoji] Function Number Percentage Reaction 334 34.3% ‘Get well soon. May you eat fast food and chocolate, Tone 247 25.3% and your sickness break, ok?’ Mention 178 18.3% Riff 95 9.7% Action 66 6.8% 17) [EmojiXpress; Emoji stickers thread; emoji] Sequence 24 2.5% Other 24 2.5% Ambiguous 7 0.7% ‘Fuck you up there (who are complaining); zip it, ok?’ Total 975 100.1%

A few graphicons in our data were ambiguous, in the sense that they could have multiple distinct 7 A pointing finger emoji was also sometimes used in this deictic function.

2189 Table 4. Graphicon function by graphicon type

Function Emoji Emoticon Image Sticker Video GIF Totals Action 55 (8.2%) 5 (4.3%) 2 (2.2%) 4 (5.3%) 0 (0.0%) 0 (0.0%) 66 (6.8%) Ambiguous 6 (0.9%) 0 (0.0%) 0 (0.0%) 1 (1.3%) 0 (0.0%) 0 (0.0%) 7 (0.7%) Riff 6 (0.9%) 3 (2.6%) 61 (65.6%) 8 (10.5%) 16 (94.1%) 1 (50.0%) 95 (9.7%) Mention 142 (21.1%) 1 (0.9%) 13 (14.0%) 22 (28.9%) 0 (0.0%) 0 (0.0%) 178 (18.3%) Other 14 (2.1%) 2 (1.7%) 6 (6.5%) 0 (0.0%) 1 (5.9%) 1 (50.0%) 24 (2.5%) Reaction 215 (32.0%) 68 (59.1%) 10 (10.8%) 41 (53.9%) 0 (0.0%) 0 (0.0%) 334 (34.3%) Sequence 24 (3.6%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 24 (2.5%) Tone 210 (31.3%) 36 (31.3%) 1 (1.1%) 0 (0.0%) 0 (0.0%) 0 (0.0%) 247 (25.3%) Total 672 (101.1%) 115 (99.9%) 93 (100.1%) 76 (99.9%) 17 (100%) 2 (100%) 975 (100.1%)

The distribution of graphicon types in the data is Overall graphicon use also varies according to shown in Table 3. Emoji were used the most by far. group category (cf. Table 1), as Table 5 shows. Use ASCII emoticons were a distant second, and the other was densest in the two Emoji groups and sparsest in types each accounted for less than 10 percent of the the Meme groups. The GIF groups patterned like the total graphicon use. Meme groups, while the Sticker groups inclined in the direction of the Emoji groups. Table 3. Frequency of graphicon types Table 5. Graphicons per comment by group category Type Number Percentage Emoji 672 68.9% Graphicons Comments Graphicons Emoticon 115 11.8% per Comment Image 93 9.5% Emoji 360 554 0.65 Sticker 76 7.8% Sticker 79 190 0.42 Video 17 1.7% GIF 272 911 0.30 GIF 2 0.2% Meme 264 1233 0.21 Total 975 99.9% Total 975 2888 0.34

Table 4 shows the distribution of graphicon The two densest categories, Emoji and Sticker, had functions by graphicon type. The most common prompts that introduced new graphicon sets and that functions (in boldface) expressed by emoji were attracted many ‘mentions’ of members of the sets in reaction, tone modification, and mention; emoji were the subsequent comments. This appears to account for also used more than the other types for actions and the higher density of graphicon use in those categories. sequences. Emoticons most often expressed reactions, It was not the case that the graphicons used in each followed by tone, whereas stickers mainly expressed group were mainly the same type as the group’s focus, reactions, followed by mentions. In contrast, images as might have been expected. Emoji were indeed most and videos mostly functioned as riffs. Our small common in the two Emoji groups, and stickers were sample of GIFs (n=2) prevents us from drawing found most often in the Sticker groups. However, conclusions about their functions. emoji, emoticons, and images were distributed Table 4 also shows that videos are the only relatively equally in the GIF groups, and emoji were graphicon type that express only one function (riff). At favored over all other types in the Meme groups, where least two graphicon types express each function, with one might have expected images to be preferred. one exception: Sequence is expressed only by emoji. All six types of graphicons can function as riffs. 4.3. Graphicons in Conversational Interaction These findings provide justification for our decision to consider the different visual elements as part of an In addition to their pragmatic functions in overarching phenomenon. They show that while the comments, graphicons can also function as functional profiles of the individual graphicon types conversational turns in and of themselves, conveying are relatively distinct, the types overlap considerably in propositional content. Of the 975 graphicons in our function. In this sense, they form part of an interrelated sample, 45.9% appeared by themselves, with no text. ecology of visual communicative elements. This occurred in all the functions, with the exception of tone marking, which accompanies text by definition.

2190 Stand-alone reactions and actions such as those in 5. Discussion examples 4-6 and example 15 function as turns in response to a prompt. In other cases, stand-alone 5.1. Research Question Revisited graphicons respond to comments by other users in interactive exchanges. In example 20, User7 uses a Our research questions asked how often and in smirking emoji following User6’s comment that the cat what functions graphicons are used in Facebook in the prompt is like User7’s cat Tinker. User7’s emoji comment threads in groups devoted to graphical was coded as a reaction. But it also functions as a turn content. The relative frequencies of occurrence of the at the interactional level, communicating a proposition graphicon types can be represented as a hierarchy, with to the effect: ‘I am wryly amused by your suggestion.’ emoji most frequent and GIFs least frequent.

Emoji > Emoticon > Image > Sticker > Video > GIF 20) [Grumpy Cats; Sourpuss thread; emoji] It is unsurprising that GIFs were rarely used, since the Facebook interface does not yet support their inclusion in comments. The low relative frequencies of occurrence of images, stickers, and videos, even in groups devoted to graphical content, are perhaps more surprising; they confirm our initial impression that use of multiple graphicon types is not yet common in Facebook comment threads. As regards functions, the graphicons in our data were used most often to react to something, usually the initial prompt of the thread.8 The relative frequencies In example 21, User9 posted “But waffles are shit, of the functions can be represented as a hierarchy, with man,” to which User8 responded with an image that reactions most frequent and sequences least frequent. contains text. ‘Nah, son!’ is a popular internet expression used for negation; the image is also a Reaction > Tone > Mention > Riff > Action > Sequence reverse image of the logo for the Hanson waffle company. This instance was coded as an action (of Each graphicon tends to specialize for function: speaking, made explicit by the inclusion of the text in a emoticons are mostly used for reactions; stickers are speech bubble). In addition to being a clever used most in reactions, then mentions; and images and intertextual reference, it functions as a conversational videos are used most to riff. In contrast, emoji express move expressing disagreement with the previous turn. all of the functions, especially tone and reaction. The emoji results partially align with previous research on 21) [Nihlist Memes; Pancakes thread; image] emoticons [4, 12, 14], which found that emoticons express tone modification and emotional reactions. But the present findings go beyond previous findings in that they identify riffing and narrative sequences, for example, as functions of emojis, but not of emoticons. The association of videos and, especially, images with the function of riffing is also consistent with observations in previous studies [10, 29]. At the same time, the fact that other graphicon types are also used to riff shows that videos and images are part of a larger graphical communication system.

5.2. Explaining Variation in Graphicon Use

What accounts for the variation we found in graphicon use? At the level of the thread, we observed that different prompts triggered different kinds of responses. Prompts containing cats were more likely than other prompts to be responded to with personal

8 Facebook’s ‘reaction’ emoji function similarly, although there are much fewer of them.

2191 photos and/or positive reaction emoji (ex. 2), for Historical and social factors also play an example. Threads that announced new graphicon sets, explanatory role. Emoji are well-established on as in the Emoji and Sticker groups, attracted mentions numerous social media platforms, whereas stickers are of those graphicons (ex. 3). Prompts that asked relatively new and specific (so far) to Facebook on web questions (e.g., the Mug prompt in the Smiley group, platforms,10 and GIFs are not currently enabled in which asked ‘Who makes your life beautiful?’) tended Facebook comments. The variation in usage of the to receive comments tagging other users with reaction, same graphicon type across Facebook groups also action, or tone emoji (ex. 14).9 Finally, prompts that appears to reflect different user demographics and to referenced subcultural knowledge (such as those in express in-group identification. For example, Nihilist Anime GIFs and Nihilist Memes) generated more Memes and Grumpy Cat Memes are both Meme riffing comments (exx. 7-9) than did other prompts. groups, but the members of Nihilist appeared more More general explanatory factors can also be sophisticated (and possibly older and more educated) invoked. Structural factors such as size and dynamicity than those of the other group. Emoji, a relatively less suggest that the graphicon types can be arranged in a complex graphicon type, are mainly used in Grumpy hierarchy with videos as the most complex and Cats, whereas the graphicon types used on Nihilist are emoticons as the least complex: more complex and varied (e.g., exx. 7-9 and 21).

Videos > GIFs > Images > Stickers > Emoji > Emoticons 6. Conclusions Graphicon complexity can also be mapped on a two- dimensional grid, with a third dimension, duration of 6.1 Implications movement, indicated by font size, as in Figure 1. We have shown that the graphicon types analyzed here are part of a larger ecology of visual communication devices that share functions, at the same time that they specialize for certain functions more than others. This specialization is due to a variety of factors, including the graphicons’ complexity, social history, and technical ease of use; the current greater popularity and functional range of emoji within that ecology can be similarly explained. An implication of this finding is that as other graphicon types become more familiar and accessible, their frequency and range of functions are likely to increase, reducing the functional space taken up by emoji, similar to how emoji have taken over many of the uses of emoticons. Our analysis also suggests that, all else being equal, the complexity (size, dynamicity, and duration) of a graphicon is associated with the frequency of its use. This suggestion could be tested experimentally. If supported, it would have implications for the design of Figure 1. A map of graphicon complexity. Duration of graphicons by sites such as Facebook that seek to movement is indicated by font size. encourage graphicon use. More generally, this study provides further The complexity hierarchy is roughly an inversion of evidence [cf. 2, 5, 6, 15, 18] that users are disposed to the frequency hierarchy in section 5.1, except that adapt graphical means – even those originally intended emoticons, stickers, and GIFs are less frequent than for unrelated purposes such as entertainment or general their complexity would predict. It may be that emoji, amusement, for example videos and text-in-image which are small, relatively static, and lacking in detail, memes – as conversational devices. It follows from this but which are not as minimalist as emoticons, hit a that other kinds of graphics, such as three-dimensional sweet spot – they are neither too large nor too detailed, representations, may well be similarly adapted in and thus lend themselves to a wide array of uses. future computer-mediated communication.

9 In such cases, the commenter often appeared to be simultaneously reacting to the prompt and orienting to their Facebook friend 10 Stickers can be used on mobile platforms via applications such as addressed in the tag. LINE.

2192 6.2. Limitations needed to reduce the likelihood of misunderstandings resulting from ambiguous graphicons. Facebook has In order to have sufficient data for this study, we tried to address the inherent ambiguity of individual analyzed only threads in groups focused on graphicons, emoji by adding text labels to their ‘reaction’ emoji, but an obvious next step would be to apply similar although it remains to be discovered to what extent methods of analysis to other Facebook threads and users actually employ ‘reactions’ in their labeled messages in other social media platforms to determine meanings. It would be possible to attach labels to all if the functions identified here capture more general emoji (and all graphicons); indeed, a number of graphicon use. Also, the threads we analyzed were graphicons already include text overlays, such as public, which enabled us to access and collect them stickers depicting cute animals with the words ‘thank easily, but it remains an open question how graphicons you’ or ‘good night,’ and animated GIFs with text are used in private conversations. indicating what the person in the GIF is saying. Even It was beyond the scope of this study to investigate videos on social media are increasingly using text cultural variation in graphicon use, although as it overlays to repeat or summarize the video’s content. happened there were many non-English comments in At the same time, some degree of ambiguity is the threads we analyzed. Research into such variation inherent in communication and is not necessarily is needed to determine whether the pragmatic functions undesirable; as [13] observes, ambiguous graphicons we identified are similar or different for different can facilitate a more fluid kind of conversation that is cultural linguistic groups, and what graphicons are “shaped by the different parties as it [goes] along.” preferentially used to fulfill those functions. This may be desirable in some situations, such as in A challenge in graphicon research is that the fliratious communication, which is common between intended meaning of some instances of use depends on social media users. A more flexible alternative to fixed personal or social context that is inaccessible to a labeling would be to allow users to attach text of their researcher considering only manifest content. In this choosing to graphicons on a use-by-use basis. study, we drew on the discourse context in which each Second, our analysis suggests that the size of some instance was embedded to identify the most plausible graphicons limits their usage. If the larger graphicons interpretation, but it would be useful to interview could appear on the same line as text, or if multiples of Facebook users about their graphicon use as a more complex graphicons (see Figure 2) could appear complement to the analysis of threads carried out here. in the same comment, it would likely expand their These interpretive difficulties were compounded by range of functions, e.g., to include tone marking and inconsistencies in the ways graphicons are rendered. sequences, which are currently expressed mostly by The Facebook interface did not consistently display all (smaller) emoji and emoticons. Relatedly, some posts or comments over multiple visits or on different stickers convey relatively simple information; they platforms. One of the authors, a Windows user, could would likely be used more if their size were reduced or never see some emoji that displayed for the other scalable to better match their complexity. author, a Mac user. We also found some emoji to be Finally, the frequency of a graphicon type is ambiguous, even when they rendered the same for both affected by its ease of use on various web-based and of us; an example is the wide-grinning/grimacing emoji mobile interfaces, as is evident from the paucity of in ex. 2, which one author sees as happy and the other GIFs in the comments we collected. We recommend author sees as angry.11 We took screenshots of threads that Facebook support users inserting GIFS in public from a single computer and coded the screenshots as a comments. This is likely to happen eventually, since way to reduce these ambiguities, but some inherent GIFs are already available in Facebook messaging. ambiguities remained. 6.4. Future Outlook 6.3. Recommendations for Graphicon Design In this paper we presented a descriptive snapshot of The problems identified above, as well as those graphicon use in Facebook comments in graphics-rich reported for emoji in [16], suggest several design groups in the first half of 2016. The incorporation of improvements that might be made to graphicons and visual elements into digital conversations is a relatively the platforms that support them. Obviously, graphicons recent phenomenon; all indications are that it will should render reliably and consistently across continue to expand in popularity, on Facebook as well computing platforms. Solutions also appear to be as in other digital media. This expansion will necessitate follow-up study by scholars of digital 11 Since the semantics of individual graphicons was not part of our media, including contextualized research that employs analysis, this did not prove to be a serious problem for our study, discourse and conversation analysis. fortunately.

2193 7. References [15] D.W. McDonald, “Visual Conversation Styles in Web Communities”, Proceedings of the 40th Hawaii International [1] N. Amaghlobeli, “Linguistic Features of Typographic Conference on System Sciences (HICSS-40), 2007. Emoticons in SMS Discourse”, Theory and Practice in [16] H. Miller, J. Thebault-Spieker, S. Chang, L. Johnson, L. Language Studies, 2(2), 2012, pp. 348-354. Terveen, and B. Hecht, “Blissfully Happy or Ready to Fight: [2] E.E. Bourlai, and S.C. Herring, “Multimodal Varying Interpretations of Emoji”, AAAI, 2016, retrieved Communication on Tumblr: ‘I Have So Many Feels!’”, June 12, 2016 from http://www-users.cs.umn.edu Proceedings of WebSci ’14: The 6th Annual ACM Web /~bhecht/publications/ICWSM2016_emoji.pdf Science Conference. Bloomington, IN, 2014, retrieved June [17] R.M. Milner, “Pop Polyvocality: Internet Memes, Public 12, 2016 from http://ella.slis.indiana.edu/~herring/tumblr.pdf Participation, and the Occupy Wall Street Movement”, [3] D. Derks, A.E.R. Bos, and J.V. Grumbkow, “Emoticons International Journal of Communication, 7, 2013, pp. 2357- and Online Message Interpretation”, Social Science 2390. Computer Review, 26(3), 2008, pp. 379-388. [18] L.N. Petersen "Sherlock Fans Talk: Mediatized Talk on [4] E. Dresner, and S.C. Herring, “Functions of the Tumblr." Northern Lights, 12, 2014, pp. 87-104. Nonverbal in CMC: Emoticons and Illocutionary Force”, [19] S. Pihlaja, “Cops, Popes, and Garbage Collectors: Communication Theory, 20, 2010, pp. 249-268. Metaphor and Antagonism in an Atheist/Christian YouTube [5] Y. Chen, F. Bentley, C. Holz, and C. Xu, “Sharing (and Video Thread”, Language@Internet, 8, 2011, article 1. Discussing) the Moment: The Conversations that Occur [20] R.R. Provine, R.J. Spencer, and D.L. Mandell, Around Shared Mobile Media”, Proceedings of MobileHCI- “Emotional Expression Online: Emoticons Punctuate 17, 2015, pp. 264-273. Website Text Messages”, Journal of Language and Social [6] M. Garau, J. Poisson, S. Lederer, and C. Beckman, Psychology, 26(3), 2007, pp. 299-307. “Speaking in Pictures: Visual Conversation Using Radar”, [21] W.V. Quine, Mathematical Logic, Norton, New York: PICS Workshop, Ubicomp, 2006, retrieved June 12, 2016 1940. from http://groups.ischool.berkeley.edu/pics/papers/Garau_ Poisson_Lederer_Beckmann_radar_pics06.pdf [22] L. Shifman, “An Anatomy of a YouTube Meme”, New Media & Society, 14(2), 2011, pp. 187–203. [7] T. Ganster, S.C. Eimler, and N.C. Krӓmer, “Same but Different!? The Differential Influence of Smilies and [23] L. Shifman, “Memes in a Digital World: Reconciling Emoticons on Person Perception”, CyberPsychology, with a Conceptual Troublemaker”, Journal of Computer- Behavior & Social Networking, 15(4), 2012, pp. 226-230. Mediated Communication, 18, 2013, pp. 362–377. [8] S.C. Herring, “Computer-Mediated Discourse Analysis: [24] N. Stockton, “Emoji – Trendy Slang or a Whole New An Approach to Analyzing Online Behavior”, in S.A. Barab, Language?”, Wired Magazine, June 24, 2015, Retrieved June R. Kling, and J.H. Gray (Eds.), Designing for Virtual 7, 2016 from http://www.wired.com/2015/06/emojitrendy- Communities in the Service of Learning, Cambridge slang-whole-new-language/ University Press, New York, 2004, pp. 338-376. [25] J. Suler, “Image, Word, Action: Interpersonal Dynamics [9] K. Inkpen, A. Roseway, A. Hoff, P. Johns, and H. Du, in a Photo-Sharing Community”, Cyberpsychology & “Video Kids: Augmenting Close Friendships with Behavior, 11(5), 2008, pp. 555-560. Asynchronous Video Conversations in VideoPal”, [26] M. Thelwall et al., “Chatting through Pictures? A Proceedings of CHI30, 2012, retrieved June 12, 2016 from Classification of Images Tweeted in One Week in the UK http://research.microsoft.com/pubs/162215/VideoPal2 and USA”, Journal of the Association for Information _chi2012_camera_ready.pdf Science and Technology, In Press. [10] J.K. Katz, and E.L. Crocker, “Selfies and Photo [27] I. Vandergriff, “A Pragmatic Investigation of Emoticon Messaging as Visual Conversation: Reports from the United Use in Nonnative/Native Speaker Text Chat”, Language States, United Kingdom and China”, International Journal of @Internet, 11(4), 2014, retrieved June 12, 2016 from Communication, 2015, pp. 2387-2396. http://www.languageatinternet.org/articles/2014/vandergriff/ [11] M. Knobel, and C. Lankshear, A New Literacies [28] A. Voida, and E.T. Mynatt, “Six Themes of the Sampler, 2007, New York, NY: Peter Lang. Communicative Appropriation of Photographic Images”, [12] T. Lanchantin, A. Simoës-Perlant, and P. Largy, “Digital Proceedings of The Twenty-third Annual ACM Conference Writing in Instant Messaging: Closer to the Oral Code than to on Human Factors in Computing Systems (CHI'05), Portland, the Written Code”, PsychNology Journal, 10(3), 2012, pp. OR, 2005, pp. 171-180. 187-214. [29] M. Wen, N. Baym, O. Tamuz, J. Teevan, S. Dumais, [13] S.S. Lim, “On Stickers and Communicative Fluidity in and A. Kalia, “Omg Ur Funny! Computer-Aided Humor with Social Media”, Social Media + Society, 2015, pp. 1-3. an Application to Chat”, Proceedings of the Sixth International Conference on Computational Creativity, IEEE, [14] S.K. Lo, “The Nonverbal Communication Functions of 2015, retrieved June 12, 2016 from http://computational Emoticons in Computer-Mediated Communication”, creativity.net/iccc2015/proceedings/4_3Wen.pdf Cyberpsychology & Behavior, 11(5), 2008, pp. 595-597.

2194